{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Convolutional Neural Networks (CNN) - 2\n",
    "- Objective: try different structures of CNNs\n",
    "- Note: examples are performed on **i5 7600 + gtx 1060 6GB **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN for Sentence Classification\n",
    "- It is widely known that CNNs are good for snapshot-like data, like images\n",
    "- However, CNNs are effectve for NLP tasks as well\n",
    "- For more information, refer to:\n",
    "    - Kim 2014 (http://emnlp2014.org/papers/pdf/EMNLP2014181.pdf)\n",
    "    - Zhang et al 2015 (https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf)\n",
    "    \n",
    "<br>\n",
    "- In this section, we perform sentence classification with CNNs (Kim 2014)\n",
    "</br>\n",
    "<img src=\"http://d3kbpzbmcynnmx.cloudfront.net/wp-content/uploads/2015/11/Screen-Shot-2015-11-06-at-8.03.47-AM.png\" style=\"width: 800px\"/>\n",
    "\n",
    "<br>\n",
    "- Pixels are made of embedding vectors of each word in a sentence\n",
    "- Convolutions are performed based on word-level\n",
    "- Classify each sentence as positive (1) or negative (0)\n",
    "\n",
    "<img src=\"http://d3kbpzbmcynnmx.cloudfront.net/wp-content/uploads/2015/11/Screen-Shot-2015-11-06-at-12.05.40-PM.png\" style=\"width: 600px\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from keras.datasets import imdb\n",
    "from keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Dataset\n",
    "- IMDb Movie reviews sentiment classification Dataset\n",
    "- Doc: https://keras.io/datasets/\n",
    "- Parameter description\n",
    "    - num_features: number of words to account for (i.e., only frequent n words are considered)\n",
    "    - sequence_length: maximum number of words for a sentence (if sentence is too short, pad by zeros)\n",
    "    - embedding_dimension: dimensionality of embedding space (i.e., dimensionality of vector representation for each word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_features = 3000\n",
    "sequence_length = 300\n",
    "embedding_dimension = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = num_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = pad_sequences(X_train, maxlen = sequence_length)\n",
    "X_test = pad_sequences(X_test, maxlen = sequence_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 300)\n",
      "(25000, 300)\n",
      "(25000,)\n",
      "(25000,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Basic CNN sentence classificationmodel\n",
    "- Basic CNN using 1D convolution and pooling\n",
    "- Known as \"temporal convolution\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation, Conv1D, MaxPooling1D, Embedding, Flatten\n",
    "from keras import optimizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def imdb_cnn():\n",
    "    model = Sequential()\n",
    "    \n",
    "    # use Embedding layer to create vector representation of each word => it is fine-tuned every iteration\n",
    "    model.add(Embedding(input_dim = 3000, output_dim = embedding_dimension, input_length = sequence_length))\n",
    "    model.add(Conv1D(filters = 50, kernel_size = 5, strides = 1, padding = 'valid'))\n",
    "    model.add(MaxPooling1D(2, padding = 'valid'))\n",
    "    \n",
    "    model.add(Flatten())\n",
    "    \n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dense(1))\n",
    "    model.add(Activation('sigmoid'))\n",
    "    \n",
    "    adam = optimizers.Adam(lr = 0.001)\n",
    "    \n",
    "    model.compile(loss='binary_crossentropy', optimizer=adam , metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = imdb_cnn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5min 51s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, y_train, batch_size = 50, epochs = 100, validation_split = 0.2, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl81fWd7/HXJycbkIWwQwBBRQkgCKSIWrdaLbRal7pg\n61Sdtlw7ttqZTu/QzuOO7XR6x8dcx7GdsTpqrZ2Oy1Xr1l5cK446LmURkVUREZIIBJQkQJKzfe8f\n35PkZDnJIQuH/M77+XjkkZzfdr7fk+R9vuf7+/6+P3POISIi2SMn0wUQEZEjS8EvIpJlFPwiIllG\nwS8ikmUU/CIiWUbBLyKSZRT8IiJZRsEvIpJlFPwiIlkmN9MF6MqoUaPclClTMl0MEZFBY/Xq1Xud\nc6PT2faoDP4pU6awatWqTBdDRGTQMLOP0t1WXT0iIllGwS8ikmUU/CIiWeao7OPvSiQSoaqqiqam\npkwXJRAKCwuZOHEieXl5mS6KiBxhgyb4q6qqKC4uZsqUKZhZposzqDnn2LdvH1VVVUydOjXTxRGR\nI6zHrh4zu8/M9pjZ+hTrzcx+YWZbzWydmc1LWrfIzLYk1i3rS0GbmpoYOXKkQr8fmBkjR47UpyeR\nLJVOH//9wKJu1i8GpiW+lgJ3AphZCLgjsX4GcJWZzehLYRX6/UevpUj26rGrxzn3iplN6WaTi4D/\ncP4ejm+a2XAzGw9MAbY657YBmNnDiW039rXQ/SUedzRGYkRjcaJxR6zjbSgdOCCod6esb4xw2/Nb\nMl0MOULyQjn82anHMHxoflrb7/zkEO/tbmDfgTB7DzbTFI4NcAllaEEu15913IA/T3/08ZcDO5Me\nVyWWdbX8lFQHMbOl+E8MTJ48uR+K1b24c2zbe5BD4Wha29fX1fHMk49y5TXfPKznueHrl/OP/3ov\nJaWlKbe549b/zfxTTmPhGWcf1rH7qqEpyr+u2NnzhhIIzkFBXg5Lz0wdLOFonOc37uKhP+3gv7fu\na7dOHxIH3qiigkET/P3COXc3cDdAZWXlgLexd9c3cSgcZcLwIQzLzyU3ZOSY0e5v28Dw3SLbo/t5\n6qH7+dnf/nW740SjUXJzU7+Mr770Qo9l+ffb/6l3leijTQ1D+PAfv5SR55Yj78x/WsGaj/anXL/3\nQDNX3PUG2/YepHz4EL5/3gl8dtooRhUVMKqogCH5oSNYWhlI/RH81cCkpMcTE8vyUizPuIamCLUN\nzYwYls+oooK09lm2bBkffPABJ598Mnl5eRQWFlJWVsbmzZt57733uPjii9m5cydNTU3cdNNNLF26\nFGibfuLAgQMsXryYz372s7z++uuUl5fz1FNPMWTIEK699louuOACLrvsMqZMmcI111zD73//eyKR\nCI8++ijTp0+ntraWr371q9TU1HDqqafywgsvsHr1akaNGjWQL5UEyLzJw3n9g3045zqd4zkUjvKN\n36yipq6Ru66ez3kzxhLKURM/qPoj+J8GvpPowz8FqHPOfWxmtcA0M5uKD/wlwFf74fn4ye83sLGm\nvlf7OqAxHMMMhuS1tWBmTCjh5gtnptzvlltuYf369axdu5aXX36ZL33pS6xfv751OOR9993HiBEj\naGxs5DOf+Qxf+cpXGDlyZLtjvP/++zz00EPcc889XHHFFfzud7/j6quv7vRco0aNYs2aNfzyl7/k\n1ltv5d577+UnP/kJn/vc5/jhD3/Is88+y69+9ate1V+y19zJZTy5toaauibKhw9pXR6LO258aC3r\nqvZz19Xz+cLMcRkspRwJ6QznfAh4AzjRzKrM7Btmdr2ZXZ/YZDmwDdgK3AP8BYBzLgp8B3gO2AQ8\n4pzbMAB1OCzhaByHoyC3bxctL1iwoN0Y+F/84hfMmTOHhQsXsnPnTt5///1O+0ydOpWTTz4ZgPnz\n57N9+/Yuj33ppZd22ua1115jyZIlACxatIiysrI+lV+yz7zJ/m9mzUeftlv+0z9s5MVNu/nxhTMV\n+lkinVE9V/Ww3gE3pFi3HP/G0K+6a5n35L3dDeSHcpgyalifyjBsWNv+L7/8Mi+++CJvvPEGQ4cO\n5eyzz+5yjHxBQVu3UigUorGxsctjt2wXCoWIRtM7+SzSk+njiynMy2HNjk+5cM4EAKr3N3L/69u5\neuFkrjltSmYLKEdM1s3VE4nFyQsdft9lcXExDQ0NXa6rq6ujrKyMoUOHsnnzZt58882+FrOT008/\nnUceeQSA559/nk8//bSHPUTaywvlMHvicNbsaDvB+4d3agD41hnHZqpYkgFZFfzxuCMWd+SFDr/a\nI0eO5PTTT2fWrFn84Ac/aLdu0aJFRKNRKioqWLZsGQsXLuyvIre6+eabef7555k1axaPPvoo48aN\no7i4uN+fR4Jt3uQyNtbU0RTxY/KffqeGOZOGc8zIvn0ClsHlqBnOeSREYnEAcnsR/AAPPvhgl8sL\nCgp45plnulzX0kc/atQo1q9vm/Xir/+6bVjo/fff32l7gMrKSl5++WUASktLee6558jNzeWNN95g\n5cqV7bqORNIxd/JwIjHH+uo6yobls6Gmnv91QZ8uqJdBKLuCP+4vD+hNV0+m7dixgyuuuIJ4PE5+\nfj733HNPposkg1DrCd4dn3Kw2Y9uu2D2+AyXSo60rAr+aKLF35uunkybNm0ab7/9dqaLIYPc6OIC\nJo0YwpqP9vPe7gYWTh3J2JLCTBdLjrDBl4B9EGkN/sHX4hfpL/Mml/HSlj1s23uwdXSPZJcsC35H\njhmhnKyqtkg7cycNJxyNk5tjLJ6lcfvZKKsS0A/lzKoqi3Qy7xjfz3/mCaMpG5beTJ0SLFnVxx+J\nOXXzSNarGF/CeTPG8uen6+5r2Sqrmr/RI9jiLyoqAqCmpobLLrusy23OPvtsVq1a1e1xbr/9dg4d\nOtT6+Itf/CL796eeYVGkJ3mhHO75eiWnHjey540lkLIm+J1zRGKO3CPc4p8wYQKPPfZYr/fvGPzL\nly9n+PDh/VE0EclSWRP80bjD0burdsFPy3zHHXe0Pv7xj3/MP/zDP3Duuecyb948TjrpJJ566qlO\n+23fvp1Zs2YB0NjYyJIlS6ioqOCSSy5pN1fPt7/9bSorK5k5cyY333wz4Cd+q6mp4ZxzzuGcc84B\n/DTPe/fuBeC2225j1qxZzJo1i9tvv731+SoqKvjWt77FzJkzOf/881POCSQi2Wlw9vE/swx2vXtY\nu+Q4x7HhGIV5OdDVqJ5xJ8HiW1Luf+WVV/K9732PG27w89E98sgjPPfcc9x4442UlJSwd+9eFi5c\nyJe//OWU97O98847GTp0KJs2bWLdunXMm9d6X3p+9rOfMWLECGKxGOeeey7r1q3jxhtv5LbbbmPF\nihWd5t1fvXo1v/71r3nrrbdwznHKKadw1llnUVZWlvb0zyKSnbKmxe8SN87t7U3G586dy549e6ip\nqeGdd96hrKyMcePG8aMf/YjZs2fz+c9/nurqanbv3p3yGK+88kprAM+ePZvZs2e3rnvkkUeYN28e\nc+fOZcOGDWzc2P2tiV977TUuueQShg0bRlFREZdeeimvvvoqkP70zyKSnQZni7+blnkqdQeaqd7f\nSMX4EkK97O65/PLLeeyxx9i1axdXXnklDzzwALW1taxevZq8vDymTJnS5XTMPfnwww+59dZbWbly\nJWVlZVx77bW9Ok6LdKd/FpHslDUt/kjMYRi5fbid3JVXXsnDDz/MY489xuWXX05dXR1jxowhLy+P\nFStW8NFHH3W7/5lnntk60dv69etZt24dAPX19QwbNozS0lJ2797dbsK3VNNBn3HGGTz55JMcOnSI\ngwcP8sQTT3DGGWf0um4ikj0GZ4u/FyKxOLkh63VXD8DMmTNpaGigvLyc8ePH87WvfY0LL7yQk046\nicrKSqZPn97t/t/+9re57rrrqKiooKKigvnz5wMwZ84c5s6dy/Tp05k0aRKnn3566z5Lly5l0aJF\nTJgwgRUrVrQunzdvHtdeey0LFiwA4Jvf/CZz585Vt46I9Mha+r6PJpWVla7j+PZNmzZRUVHR62Nu\nqz1A3MHxY4r6WrzA6OtrKiJHDzNb7ZyrTGfbrOnqieqqXRERIIuCPxLXPD0iIjDIgr+33VKxxC0X\nj/RVu0ezo7GLT0SOjEET/IWFhezbt69XgRUZxDdgGQjOOfbt20dhoW7AIZKNBs2onokTJ1JVVUVt\nbe1h79sciVF7IEz8k3x25YUGoHSDT2FhIRMnTsx0MUQkAwZN8Ofl5TF1au+mkX18TRV/9fQ7vPT9\nszh2tEb1iEh2y4q+j131/irYcaXq2hARyYrg313XRElhLkPzB80HHBGRAZMVwb+rvkmtfRGRhKwI\n/t31zYwtUfCLiECWBP+nh8KM0E2lRUSALAn+usYIpUPyMl0MEZGjQuCDPx531Cv4RURaBT74D4Sj\nxB0KfhGRhMAHf92hCAAlCn4RESAbgr/RB79a/CIiXlrBb2aLzGyLmW01s2VdrC8zsyfMbJ2Z/cnM\nZiWt225m75rZWjNb1XHfgVav4BcRaafHS1nNLATcAZwHVAErzexp59zGpM1+BKx1zl1iZtMT25+b\ntP4c59zefix32tTiFxFpL50W/wJgq3Num3MuDDwMXNRhmxnASwDOuc3AFDMb268l7aWW4Fcfv4iI\nl07wlwM7kx5XJZYlewe4FMDMFgDHAC1z/jrgRTNbbWZL+1bcw6cWv4hIe/01a9ktwM/NbC3wLvA2\nEEus+6xzrtrMxgAvmNlm59wrHQ+QeFNYCjB58uR+KpYP/lCOMSxf8/CLiEB6Lf5qYFLS44mJZa2c\nc/XOueuccycDXwdGA9sS66oT3/cAT+C7jjpxzt3tnKt0zlWOHj36sCuSSn2Tv3jLTLddFBGB9IJ/\nJTDNzKaaWT6wBHg6eQMzG55YB/BN4BXnXL2ZDTOz4sQ2w4DzgfX9V/ye1TVG1c0jIpKkx64e51zU\nzL4DPAeEgPuccxvM7PrE+ruACuA3ZuaADcA3EruPBZ5ItLZzgQedc8/2fzVSq2uM6MSuiEiStPr4\nnXPLgeUdlt2V9PMbwAld7LcNmNPHMvaJJmgTEWkv8FfuaoI2EZH2Ah/8vsWvWy6KiLQIdPA759TV\nIyLSQaCD/2A4RizuKClU8IuItAh08OuqXRGRzoId/IcU/CIiHQU7+NXiFxHpJCuCXxdwiYi0CXTw\n1zepxS8i0lGwg7+lq2eogl9EpEWgg7+uMUKOQVG+LuASEWkR+OAvGZJHTo6mZBYRaRH84NfFWyIi\n7QQ++HViV0SkPQW/iEiWUfCLiGSZQAd/ve6+JSLSSWCDX1Myi4h0LbDB3xiJEYk5Bb+ISAeBDX5N\n0CYi0rXABn99YxRQ8IuIdBTY4FeLX0Ska4EP/hLdaF1EpJ3AB79a/CIi7Sn4RUSyTOCDv1iTtImI\ntBPY4K9vjFBcmEtIUzKLiLQT2ODXVbsiIl1T8IuIZBkFv4hIllHwi4hkmcAG/4GmKEUFunhLRKSj\nwAZ/OBanIC+w1RMR6bXAJmM4Gic/FMp0MUREjjrBDf5YnLxcjeEXEekoreA3s0VmtsXMtprZsi7W\nl5nZE2a2zsz+ZGaz0t13IDjnCEfjFIQC+74mItJrPSajmYWAO4DFwAzgKjOb0WGzHwFrnXOzga8D\nPz+MfftdJOYAyM9V8IuIdJROMi4AtjrntjnnwsDDwEUdtpkBvATgnNsMTDGzsWnu2+8isTgAeWrx\ni4h0kk4ylgM7kx5XJZYlewe4FMDMFgDHABPT3LffhaM++NXiFxHprL+S8RZguJmtBb4LvA3EDucA\nZrbUzFaZ2ara2to+FSYcU/CLiKSSzhVO1cCkpMcTE8taOefqgesAzMyAD4FtwJCe9k06xt3A3QCV\nlZUuveJ3raXFr64eEZHO0knGlcA0M5tqZvnAEuDp5A3MbHhiHcA3gVcSbwY97jsQWlr8BWrxi4h0\n0mOL3zkXNbPvAM8BIeA+59wGM7s+sf4uoAL4jZk5YAPwje72HZiqtGnt41eLX0Skk7Qms3HOLQeW\nd1h2V9LPbwAnpLvvQFNXj4hIaoFMxohO7oqIpBTIZNRwThGR1AKZjM1q8YuIpBTIZIzo5K6ISEqB\nTEZdwCUiklogk1HDOUVEUgtkMrZO0qYWv4hIJ4FMRrX4RURSC2QyNms4p4hISoFMxtYbsajFLyLS\nSSCTURdwiYikFshkDMdihHKMUI5uti4i0lEwgz8aJy+k0BcR6Uoggz8Sc+rfFxFJIZDp2ByNk58b\nynQxRESOSoEM/nA0rrtviYikEMh0jMTUxy8ikkoggz8cjWsop4hICoFMx3BMwS8ikkog09F39QSy\naiIifZbWzdYHm+ZonGnx7XDgBCganeni9E7zAdjwBHz8DjTXQ3MDFI2F02+EEcem3i/S5LdvqofI\nIcgfBgUlUFgKufntt43H4JNtMGwUDClLr1zxOIQbfHma6iF/KJROhpw03midg/oa2PoibH0Bdm+E\ncbNg0ikwfg7Eo/6Y4QN+WwAXh/BB/3yRgzD5NDj+XMhJGrUVj8GB3VBXDfVVkF8Eo6dD6UQwnesR\n6Sh4wR+Pc0ndb7nq0APw4tfg4l/2z3FjUdj4JGx8Cho/9UEUbYKiMVA6CYaNhkN7oa4KDu6FcSfB\n8Z+HqWfBvq3w/vOw7WUfyi1yh0BBMRSW+PAtmQglE+DjtbDuUR+whaVQONyH9wcr4O3fwslfhdlX\nwqfbYc8m2Pu+f976KmiqS12H4cfAmBlQNgVqN0HVKh+yAMXjYfSJYDm+bs0Nvmwl5T5AG/fDno1Q\nu8UHcLK8YX7fsim+LgUlgEsEcbUP5aZ6X/d41O9TUg7jT4aat/1rmg7Lgdf+xb/eJ38NXAx2vgVV\nqzuXCXw5Rk+HMRW+3gVFvkx1O9u/Tjm5bb+HUEGi/vX+95s/DApK29YXlPjjWNIbT/4w/3vKGwqH\n9vk611dDLNK5TM5B4yeJ31e1/xsYM92Xb8gIaK7zr9WhfX6buipf7+POgePP83XZt9X/3vfv8K9z\ny2uTX+TLmF/kH7csLxrjf4dFY9u/YUrWMtfSsjqKVFZWulWrVh3+jk118MT1sGU5Ycsnf/xMWPry\n4R0jHoM374T3n4MRx/l/yFgY3vp3qNvhw7l0og+C3IK2lubBPTB0FJSW+3/g6lX+DaKFhXzLtmS8\nf+ycD5bmBl/uA3v8sXCQWwgzL4H51/p9WlqtDbvg1dtg9a99mcBvO3KaL1NpuQ/wwtJEC7/Qt/qb\nG3yQ1G7xgfHph36fyafAhLl+3Z5Nfr2Zr1tBsS9XS3gXFPvXYswM/1yFJYlt6v2+ezb4kGr5JAC+\nPCUtZUqEZtEY/2Y4pqKtXvUf+/1b3giTg9XMB1l+kX+8Zbmv/7aX/TbjZsHEBf54pRP98zU3+Dep\nPRthz2b/vfGTtt/FsDEwdASQeP54pK3c0abEJ6QS//qFDyQ+PXXxxtJbBaVtr03kEOzeAE3722+T\nO6T9NtWr/aefvggVwNWPwdQz+3YcOSqZ2WrnXGVa2wYm+A99Ar86Hz7Zxh35f86MwlrOaXoJlu1I\n/+P+Jx/Ck38BO16HUScmWqqJf8hJC303ywmL0+vWiMf8P+v212DkcXDs2T6MuxMNQ8PHMGR499vW\nVcOud2HUNN/KPhpbcc4NbDdLw+5Ea7wovbIc2OMDtGSCf8Pubtuuyh2LtnVxNTe074qKHGrroho6\nou2TW96Qrp+j4/Gd839rzQ2JTxTFft/k7Q59Ah+85P9GR03zb8AjpvpPK+D/3sINnbvK4tFE46QK\nnv0hLPgWfOFnPb9mMugcTvAHp6tnSJnvWqm4gEcejfDdwpegrt7/wxePbduursr/E407qe0fK9II\nK++FFf/oQ/Tiu2DOEr+uYZf/Rxo17fDKkxOCSQv8V7py86HsmJ63Ky33X0ezge5bT/6d9sQs/e1T\nlTuU6//G0j0XcjjMoHic/0pl6Ag46bLU69Mp39oHfdeaZL3gBL8ZLL4FgHD0j3xSONkv37e1/T/9\n4/8DPnrNB//8a/3Jylf/GQ7s8m8cF9wOwye1bd/SNSMy2JXPg7cf8J8OjsZPiXLEBHLMYyQWZ//Q\nRMt53/ttK+Jxf+J04mf8ObH/93145gd+lMy1y+Hq37UPfZEgKZ/vz1XUbsl0SSTDgtPiT9IcjdNY\nOMGfzNq3tW3Fpx/6bpt5X4e5f+bfBKJh3x2jYX8SdBPm+e81a2DsjMyWRTIqkC3+cDROfl6uP6m6\n74O2Fbve9d/HzvJBP2GuH9mi0JdsMPJ4f/K4enWmSyIZFsjgb71yd+Rxfox7i93r/RDAMRWZK5xI\npuTkwISToXpNpksiGRa44I/G4sRd4n67I6f57p1Y4qKhXe/CqBNSD7MTCboJ8/x1A5GmTJdEMihw\nwR+OJd1ofeTxfhzz/o/8yl3v+tE8ItmqfL6/YG33+kyXRDIocCd3I1F/4Yrv6jneL9y31Y9vrq/2\nV3qKZKvyxAne6jUwMa1rfbKDc/5izboqf+1Py1QYJF3JXlDcfqqOZHmJq847DpNt2A0734Sdf2q7\nsr253l98OLrCdzuPSlx5XzweQnkDWctWgQv+5lgMSLT4Wy662rfVX34PavFLdisp93P2HA0neMOH\nYO8WP/quZQqQ/OK2K+Nb1u/Z7OclqtvpJ/nLG9I2PUdBceJgDqLNbXNCFZb60Xrl85O26fj8B2HL\nM34yxG0vt81b1Rf5xf5iOvBXdbfMCRUqgOGTk6Y6qYM1v/FXfbcy31j9bi+mqzlMgQv+cNR39RSE\ncvzVjkPK/AnelkvYxyr4JYuZ+X7+miNwgre5wYd27Sb49KO2uYbCB/wbz8fvtE3alyy/2M/62q7l\njX/DKpkA9Y2w7b/8FBVdyRvqr8bH+Unqiie0TVqX7GAtRBt9S3v2FX5urtJyKBrX1nJ3cT9TbnOd\n/97lfEnOv0m1tOaT6zR8sp/uZfycLmbHjcP+7X4ajpZJ+7qa2G8ApBX8ZrYI+DkQAu51zt3SYX0p\n8J/A5MQxb3XO/TqxbjvQAMSAaLpzSfRWJJbo6slNDNEcebxv8UcSv+DBOk2zSH8pnw/vPeNbnclz\nQjkH/317IgivTD3MORbxM9AWFPsuCxf3U5vUV8Pe92DHm37W1L3vte1jOW3dJKF8P7rotO/6IdUu\n3tZSb06aD6l0ou8KGV3hpzLpOMdSU50P3Ba5Bb5MoTw/m2z1Ktjxlg/VrhSWQsWFMPnU9Obf6m85\nOf7i0e6mWR8gPQa/mYWAO4DzgCpgpZk97ZzbmLTZDcBG59yFZjYa2GJmDzjnElNIco5zbm9/F74r\nLS3+/FDij2zkNP8xrvFTP35fJNuVz/Xfa9bCsWe1LX/h7+D1X/if1/wWLry9/RxVzsGGx+GFm323\nC7SFuYu1bVc43M8qe9IVMHamD+/hx/R/uLbMQtuVIcP9FCzHf75/nzMg0mnxLwC2Oue2AZjZw8BF\nQHLwO6DYzAwoAj4BuvgMN/Bag7/l1osjj4N3HvQzFE47PxNFEjm6TJjnW91/+Eu44F98+L92uw/9\nym/A+Nn+TeDO0+DExf7+ByUT/H0Tdr7lu0tP+67/FN1yf4mW+zaMONZ3mWSiBS1pSyf4y4GdSY+r\ngFM6bPNvwNNADVAMXOlca2eYA140sxjw7865u/tW5O6FEyd380JJXT3gWyQ6sSviz3197TH4/U3w\nH1/290f48L9g1lfgi7f60D5hMfzx72HHG/Dec/4+BcNGw5f/1d8ER5O8DWr9dXL3C8Ba4HPAccAL\nZvaqc64e+KxzrtrMxiSWb3bOvdLxAGa2FFgKMHny5F4XJJwYztna4k/+qDpudq+PKxIox54Ff/EG\nvPJ/4L9/7rtELr6rraVePBYuvsP/7Jy/WU9+EeQVZq7M0m/S+TxWDSRPWTkxsSzZdcDjztsKfAhM\nB3DOVSe+7wGewHcddeKcu9s5V+mcqxw9uvcnYFsu4CpoCf6WEyd5Q/2NK0TEyxsC5/4d/NVmuOr/\ndh510sLM3xpUoR8Y6QT/SmCamU01s3xgCb5bJ9kO4FwAMxsLnAhsM7NhZlacWD4MOB8Y0EsGO53c\nzRvi+yjHztTHU5GuFI1uG3suWaHH37ZzLmpm3wGeww/nvM85t8HMrk+svwv4KXC/mb2Lv5Hp3zjn\n9prZscAT/pwvucCDzrlnB6gugJ+gDZKGcwKc9/c93/ZQRCRLpPU275xbDizvsOyupJ9r8K35jvtt\nA+b0sYyHpa3Fn/RhZtalR7IIIiJHtcCNueo0nFNERNoJXDq2zs4ZClzVRET6ReDSUS1+EZHuBS4d\n283HLyIinQQuHSOJFn+eunpERLoUuHQMx+KYQW6ObqAuItKV4AV/NE5+KAdLNaWsiEiWC1zwNyeC\nX0REuha4hIzE4jqxKyLSjcAlZDiq4BcR6U7gEjKsFr+ISLcCl5CRWFxDOUVEuhG4hAzr5K6ISLcC\nl5DN6uMXEelW4BIyElOLX0SkO4FLSI3qERHpXuASUqN6RES6F7iEjEQdeSFN1yAikkrggt+3+HVT\ndRGRVIIX/BrOKSLSrcAlpB/Oqa4eEZFUAhf8Gs4pItK9wCWkhnOKiHQvcAmp4ZwiIt0LVELG4o5Y\n3GmSNhGRbgQqISMxf6N1tfhFRFILVEI2RxPBrxa/iEhKgUpItfhFRHoWqIQMq8UvItKjQCVka/Cr\nxS8iklKgErKlq0ejekREUgtUQjarxS8i0qNAJWRYJ3dFRHoUqISM6OSuiEiPApWQavGLiPQsrYQ0\ns0VmtsXMtprZsi7Wl5rZ783sHTPbYGbXpbtvf9JwThGRnvWYkGYWAu4AFgMzgKvMbEaHzW4ANjrn\n5gBnA/9sZvlp7ttvNJxTRKRn6STkAmCrc26bcy4MPAxc1GEbBxSbmQFFwCdANM19+01YwzlFRHqU\nTkKWAzuTHlclliX7N6ACqAHeBW5yzsXT3LfftLT4C9TiFxFJqb8S8gvAWmACcDLwb2ZWcjgHMLOl\nZrbKzFbV1tb2qhA6uSsi0rN0ErIamJT0eGJiWbLrgMedtxX4EJie5r4AOOfuds5VOucqR48enW75\n22kZzql2uuygAAAFSklEQVSuHhGR1NJJyJXANDObamb5wBLg6Q7b7ADOBTCzscCJwLY09+03avGL\niPQst6cNnHNRM/sO8BwQAu5zzm0ws+sT6+8Cfgrcb2bvAgb8jXNuL0BX+w5MVTScU0QkHT0GP4Bz\nbjmwvMOyu5J+rgHOT3ffgRKOOQDyQnYknk5EZFAKVNM4HI2TH8rBjyoVEZGuBC/41b8vItKtQKVk\nJKbgFxHpSaBSMhyNq39fRKQHwQp+tfhFRHoUqJRsObkrIiKpBSolw7G4rtoVEelBoFIyHI1rgjYR\nkR4EKiU1nFNEpGeBSsmIunpERHoUqJTUqB4RkZ4FKiU1qkdEpGeBSslwLE6eWvwiIt0KVEqGo3EK\n1OIXEelWoFJSo3pERHoWqJTUJG0iIj0LVEr6SdoCVSURkX4XqJQ8b8ZYZk4oyXQxRESOamndenGw\nuH3J3EwXQUTkqBeoFr+IiPRMwS8ikmUU/CIiWUbBLyKSZRT8IiJZRsEvIpJlFPwiIllGwS8ikmXM\nOZfpMnRiZrXAR73cfRSwtx+LMxhkY50hO+udjXWG7Kz34db5GOfc6HQ2PCqDvy/MbJVzrjLT5TiS\nsrHOkJ31zsY6Q3bWeyDrrK4eEZEso+AXEckyQQz+uzNdgAzIxjpDdtY7G+sM2VnvAatz4Pr4RUSk\ne0Fs8YuISDcCE/xmtsjMtpjZVjNblunyDBQzm2RmK8xso5ltMLObEstHmNkLZvZ+4ntZpsva38ws\nZGZvm9kfEo+zoc7DzewxM9tsZpvM7NSg19vM/jLxt73ezB4ys8Ig1tnM7jOzPWa2PmlZynqa2Q8T\n+bbFzL7Ql+cORPCbWQi4A1gMzACuMrMZmS3VgIkC33fOzQAWAjck6roM+KNzbhrwx8TjoLkJ2JT0\nOBvq/HPgWefcdGAOvv6BrbeZlQM3ApXOuVlACFhCMOt8P7Cow7Iu65n4H18CzEzs88tE7vVKIIIf\nWABsdc5tc86FgYeBizJcpgHhnPvYObcm8XMDPgjK8fX9TWKz3wAXZ6aEA8PMJgJfAu5NWhz0OpcC\nZwK/AnDOhZ1z+wl4vfF3BhxiZrnAUKCGANbZOfcK8EmHxanqeRHwsHOu2Tn3IbAVn3u9EpTgLwd2\nJj2uSiwLNDObAswF3gLGOuc+TqzaBYzNULEGyu3A/wTiScuCXuepQC3w60QX171mNowA19s5Vw3c\nCuwAPgbqnHPPE+A6d5Cqnv2acUEJ/qxjZkXA74DvOefqk9c5P1QrMMO1zOwCYI9zbnWqbYJW54Rc\nYB5wp3NuLnCQDl0cQat3ok/7Ivyb3gRgmJldnbxN0OqcykDWMyjBXw1MSno8MbEskMwsDx/6Dzjn\nHk8s3m1m4xPrxwN7MlW+AXA68GUz247vxvucmf0nwa4z+FZdlXPurcTjx/BvBEGu9+eBD51ztc65\nCPA4cBrBrnOyVPXs14wLSvCvBKaZ2VQzy8efBHk6w2UaEGZm+D7fTc6525JWPQ1ck/j5GuCpI122\ngeKc+6FzbqJzbgr+d/uSc+5qAlxnAOfcLmCnmZ2YWHQusJFg13sHsNDMhib+1s/Fn8cKcp2Tparn\n08ASMysws6nANOBPvX4W51wgvoAvAu8BHwB/m+nyDGA9P4v/+LcOWJv4+iIwEj8K4H3gRWBEpss6\nQPU/G/hD4ufA1xk4GViV+H0/CZQFvd7AT4DNwHrgt0BBEOsMPIQ/jxHBf7r7Rnf1BP42kW9bgMV9\neW5duSsikmWC0tUjIiJpUvCLiGQZBb+ISJZR8IuIZBkFv4hIllHwi4hkGQW/iEiWUfCLiGSZ/w8p\nPa94MJi5VAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c5a2e17940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24832/25000 [============================>.] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.8556\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 1. Advanced CNN sentence classification model - 1\n",
    "- Advanced CNN using 2D convolution and pooling\n",
    "    - Embedding layer is \"reshaped\" to 4D to fit into 2D convolutional layer\n",
    "- Perform global max pooling for each window"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers import Reshape, Conv2D, GlobalMaxPooling2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def imdb_cnn_2():\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Embedding(input_dim = 3000, output_dim = embedding_dimension, input_length = sequence_length))\n",
    "    model.add(Reshape((sequence_length, embedding_dimension, 1), input_shape = (sequence_length, embedding_dimension)))\n",
    "    model.add(Conv2D(filters = 50, kernel_size = (5, embedding_dimension), strides = (1,1), padding = 'valid'))\n",
    "    model.add(GlobalMaxPooling2D())\n",
    "\n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.3))\n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.3))\n",
    "    model.add(Dense(1))\n",
    "    model.add(Activation('sigmoid'))\n",
    "\n",
    "    adam = optimizers.Adam(lr = 0.001)\n",
    "\n",
    "    model.compile(loss='binary_crossentropy', optimizer=adam , metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = imdb_cnn_2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, y_train, batch_size = 50, epochs = 100, validation_split = 0.2, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecVPW9//HXZ2e298YCu1Sld1hRLCjBAnaMBaOJopFo\nNJp+Tfldc2+Se/P7mevVXE2IGowalXhRBBNssYDGRpHehGWB3QV2l+1lZnZmvr8/vgMsWwcYWDjz\neT4ePNg5c86Z75nyPt/zOd85I8YYlFJKRY+Ynm6AUkqpk0uDXymloowGv1JKRRkNfqWUijIa/Eop\nFWU0+JVSKspo8CulVJTR4FdKqSijwa+UUlHG3dMN6EhOTo4ZOHBgTzdDKaVOG6tWrao0xuSGM+8p\nGfwDBw5k5cqVPd0MpZQ6bYjIrnDn1VKPUkpFGQ1+pZSKMhr8SikVZbqt8YvIfOBKoNwYM7qD+wV4\nDLgcaAJuN8asDt03I3SfC3jaGPObY21oS0sLJSUleDyeY12FaiUhIYGCggJiY2N7uilKqZMsnJO7\nfwYeB57r5P6ZwJDQv7OBPwBni4gLeAK4BCgBVojIEmPMpmNpaElJCampqQwcOBC7r1HHyhjDgQMH\nKCkpYdCgQT3dHKXUSdZtqccYsxyo6mKWa4DnjPUpkCEifYDJwHZjTJExxgcsCM17TDweD9nZ2Rr6\nESAiZGdn69GTUlEqEjX+fGBPq9sloWmdTT9mGvqRo8+lUtHrlBnHLyJzgbkA/fv37+HWKNVzAkFD\n0BhiXafO2AufPwhAnPvUaVNrnpYAca4YYmIi16EJBg1ltc0EgvbnaWNE6JuRiCv0GMYYtpc3sGxb\nBUlxbgbnJjM4N5m0hFhcMYJLJKLtiaRIBH8p0K/V7YLQtNhOpnfIGPMk8CRAYWHhKfdDwDU1Nbz4\n4ot8+9vfPqrlLr/8cl588UUyMjI6nedf//VfmTp1KhdffPHxNvO0tGZPDU99WMQVY/pw2ajehz5Y\nx+Lgh7TtOpp8fooqGqlq9FHd5GNvrYcd5Q0UVTbi9Qc494wcLhiSw1kDs0iIdYX1WP5AkH11HvbW\nethX66G83kt6Yiz5GYkUZNp/rY+sNpXV8di722gJGM4ZnMWUwTmkJ8ZSUtNEWY2HbfvrWbOnhg2l\ntQSChrMHZzN1SA5j8tMx2CDyBoLUNbdQ19xC0MDo/DRG9U1v12ZjDGW1HrbsrSMjKZb8jCQyk2NZ\ntaua97eU80nRAYbmpXLdhAKmnJGNK0aobvSxdX89e6qa2F/noazWw56qJooPNFJa3UzQQE5KHHlp\nCeSlJZCTEkdOSjyuGKG0upmSmmZiBGZNyOfKsX1Jjrfx4mkJsL28gV0H7Lr21jZjjH2N3DExZCbF\nkpMaT1ZyHKnxbhLjXCTGufAHDF5/EE9LgJ2VjWzdV8/W/fU0+fyhbYQGr58DDT4avH76pidwx/mD\nmD25PynxbirqvfxzeyVVjT6G90llZJ80MpLi2j1Pu6ua2LqvnuomH1WNLeyv87CxrJZNZXU0+gJH\nzJ8Y62Jk3zQG5ySzalc1RZWNXb5HYgTcrhjiXDGkxLvJSY0jOzmeC4fmMue8njtfKeH82LqIDAT+\n1smoniuA+7Cjes4GfmeMmSwibmAbMB0b+CuArxljNnb3eIWFhabtN3c3b97MiBEjum3riVJcXMyV\nV17Jhg0bjpju9/txu0+ZA6ej0tlz6g8EqWryUVnv40CjlwMNPkQgJd5NcrybRq+f3VVN7KlqJtYt\nTOiXwYT+mRgD720p570t5RRVNOB2hT7YybGcPSib887MZmxBxqGerDGG5z/dxS//tgljwB80DM5J\n5tZzBlDd5GP17mq27mugcEAmN55VwNQhuXj9QT4vrmJlcRUDspK5bFRv0pNiafD6efbjYp76sAh3\nTAxfnZjPDYUF+PyGFz/fxWtflNHg9R+xnTkp8ZyRm4wIrN5Vgy8QJDHWxSUj87hmfF/G9cvgvc3l\nvL6ujBXFVaQmxJKdHEdaQiz76z2UVjfjD3b++emTnsClI/O4aFgvlq7fy8LVJaSF1tFRYMS5YxjZ\nJ41xBemICB9+WcGOiq6DBSDWJZzZK5X0RDdJcfa9uKG0lvJ6b4fzx7liGN8vg8376qj3+MlNjUeg\n3fxZyXHkZyQyMCeZQdlJuGJiQju6ZsrrvIfeGwbonZZAfkYiVU0+tpc3kBzn4qxBWewOhX3rpykz\nKZYYEQLG0OIPtgvXzqTGuxnaO5WMxMMj0ZLj3WSnxJGVFMdH2yv5bGcVaQlu8jOT2Ly3rt06clLi\nyUmJIzc1HoD1pbXUNLUcMU9SnIsRfdIY3TeN4X3SiA8d5fj8Qbbur2dDaS1fljcwum86l43K4+KR\nefgDhqLKRnZWNNDoCxAMGgLGEAgafIEgLX5DvaeFygYvZTUetu6v5/IxvXn4+nGHdpDHS0RWGWMK\nw5q3u+AXkZeAi4AcYD/wELY3jzFmXmg45+PADOxwzjnGmJWhZS8HHsUO55xvjPl1OI06FYN/9uzZ\nLF68mGHDhhEbG0tCQgKZmZls2bKFbdu2ce2117Jnzx48Hg8PPPAAc+fOBQ5ffqKhoYGZM2dy/vnn\n8/HHH5Ofn8/ixYtJTEzk9ttv58orr+T6669n4MCB3Hbbbbz++uu0tLTw0oK/MnzEcCorKvn6rbdQ\nVlbGlClTeOedd1j2z0+RxDSaWwIkuF0kxLpIiI0h3u0i3h2DCHhagjR4/TT5/PiD9o0YDH0K9+7a\nwc8/qAZs7ytGoLa5heo2H4TOJMa68AeDtASOfA8VZCYyriCDoDG0BAx7a5vZtLcOYw73mMbkp7O/\nzsMbG/YxfXgvHr5hHJ/sOMDvP9jOxrI6XDHC8N6pnNkrhY++rORAo4+s5DjqPS20BAwitscX6xLO\nGZzNhtJaqptamDYsl1hXDO9tKT8UyvHuGK4c25dLRvYiJyWejKQ4eqXFk5ZwOECafH4+21nFO5v2\ns3T93iPCoF9WItOG9aIlEKSi3kedp4W8tAT6ZyXSLzOJPhmJ9ElPIDclnprmFkqrmyk+0MiybRUs\n31aB1x8k1iXcfu5A7ps2hPSkWPbXefi06ADeliD5mYnkZyTSNyOxXSmltKaZooqGQ2WDWFcM6Ylu\n0hJjCQZhXUkNX+ypYcveOhq9ARp9fgJBw4g+aYzvl8HIvmnUe1oorfFQUedhVH4655+ZQ3K8G09L\ngPe2lPP39XuJd8cwvHcqQ/NSGZSTTF5aQlhHPsFQWcrdame+enc1L32+h7V7ahicm8yw3mkMzUth\nUE4yA7OT2wWd1x+gqtF3qOfe7AvQ3BLAFSMkxNr3cr+sJPqmJ3TbQ/5idzVPf7ST6kYf552Zw9Qh\nueSlx7Nlbz2b9tZRXNlIZYOXygYfgaBhdH4aYwsyGNEnjdzUeDKTYkmMdZ3Qnrgxhqc/3Ml/vrGZ\noXmp/HjGMPbWeiiqsEefv7p2zDGtN6LB3xO6C/5/e30jm8ra782Px8i+aTx01ahO72/d4//ggw+4\n4oor2LBhw6HhkFVVVWRlZdHc3MxZZ53FsmXLyM7OPiL4zzzzTD797HNGjhnLLTfP5tKZV3DdjTfz\nnbvv4uLLZnLtdddROHo4d993P7Nvn8v8p/7IpvVr+cXDv+M/fv4j8nr34e4Hfsgny97lzpuv44O1\n28nNsR9irz+ItyWI4fDr6Qr1qsD2JmNjYmztMVQG2VP0Ja/uhKAxoQ8wpCW6yU6OJzt0GJ+TYv8W\n7GF1vcdPYpyL/llJZCfH4fUH2bS3jtW7qgkaw0XDejGkV0q7D051o49Pig7w+c4qNpTWsmlvHZ6W\nAD+8bBh3Tz3jUC3UGNtz6pOecKj36vMHeW9LOW9s2Euf9ETOOzObwgFZfFlez9/W7eWdTfsZmJ3E\nAxcPZXw/W1KrqPeyZG0ZLoFrJ+S3O8Tvis8f5KPtFWwsrWPq0FzGhnrgx+LgDuXM3BT6ZSUd0zqU\nMy3bVsF3XlxNncceiSbExjCiTxqv3nPuMb3fjib4T88axSlg8uTJR4yB/93vfseiRYsA2LNnD5u3\nbGXEuEn4g4YdFQ00NNST328AcXmD2V7ewIBho9m4dQfn1Xnw+QM0ePzsq/UQCAY566IZ+IOGs88q\nZPk7f6dPeiIbVn/O95//K6nxbi6afgnpGZkUZCQyqE8aMaE3STBo8PoDdifgD+IPBEmMc5MS7+7w\npFzD/jh+e8PxHUUlxLqY2D+Tif0zu5wvMzmOy8f04fIxfQBbi/e0BNr1/kSEM3JTjpgW545hxuje\nzBjd+4jpYwsyGFuQwU8vb78Nuanx3Hn+sX1HIc4dw1eG5/GV4XnHtHxrSXFupg3rddzrUc5z4dBc\n3vn+hWwvb2BgTjJ90hJO2sng0zL4u+qZnwzGGJKSkmjwtNDoC/CP995nydK3+NOrb5KSnMw3rruC\nrWVVpA2wJ7HcMUJirJvEhAT6pNvD+by0JJqbGhmTn05GUhz9s5MY1TedWFcMY/rn0DsvlbrMZGJM\nkNzUeNwxQm5aPAWhXmOMQHpS3KHQB4iJERLj3CSG37ntMa4YiVhtU6nT1cET5SfbqTk26xSUmppK\nXV0928sbKK5spN7rp6iykfI6D3W1tWRnZdInO4Oy4h2sXb2CrKQ4hualEusSBmQn0zcjEbdLyE2N\nJz0xlli3HXrW+pDuYAnG3cEwvvPOO4+XX34ZgLfffpvq6uqTs+FKKcfRLleYktMyGDNpMjOnTiY5\nKZFevfIYlJNMYpyLobNnsWTBs1w8ZSLDhg1jyjnnkJkcF/awwHA89NBD3HzzzTz//PNMmTKF3r17\nk5qaGrH1K6Wix2l5cvdk8/mD7KhowABn5iYT545coIfL6/Xicrlwu9188skn3HPPPaxZs+a41tnT\nI6WUUpGjJ3cjyB8I2nHIQcPgHgp9gN27d3PjjTcSDAaJi4vjqaee6pF2KKVOfxr8nQgEjR3vW+8l\nCAzKTiIxrueeriFDhvDFF1/02OMrpZxDg78DDR77zVR/MEhaQiy908P7MotSSp0ONPjbsNc4aSYm\nBs7MSTn0JSKllHIKHc7ZRp3Hj6clQF5agoa+UsqRNPhbMcZQXuch3h1zxIWglFLKSTT4W2nw+mlu\nCZCb2v3FoLqTkmIvO1BWVsb111/f4TwXXXQRbYettvXoo4/S1NR06Pbll19OTU3NcbVNKRXdNPhD\njDHsr/MS54ohIylyvf2+ffuycOHCY16+bfAvXbq0y2v7K6VUdzT4QxpDly7OTY0/4vo3Bz344IM8\n8cQTh27/4he/4Fe/+hXTp09n4sSJjBkzhsWLF7dbrri4mNGj7c8YNDc3M3v2bEaMGMGsWbNobm4+\nNN8999xDYWEho0aN4qGHHgLshd/KysqYNm0a06ZNA+xlnisrKwF45JFHGD16NKNHj+bRRx899Hgj\nRozgrrvuYtSoUVx66aVHPI5SSp2eZy/feBD2rY/oKgPpw3BP+QWZnVy+96abbuK73/0u9957LwAv\nv/wyb731Fvfffz9paWlUVlZyzjnncPXVV3daJvrDH/5AUlISmzdvZt26dUycOPHQfb/+9a/Jysoi\nEAgwffp01q1bx/33388jjzzC+++/T05OzhHrWrVqFc888wyfffYZxhjOPvtsLrzwQjIzM/nyyy95\n6aWXeOqpp7jxxht55ZVXuPXWWyP0TCmlTnfa4wcCxuALGHJS4jq9LOqECRMoLy+nrKyMtWvXkpmZ\nSe/evfnpT3/K2LFjufjiiyktLWX//v2dPs7y5csPBfDYsWMZO3bsoftefvllJk6cyIQJE9i4cSOb\nNm3qss0fffQRs2bNIjk5mZSUFK677jo+/PBDAAYNGsT48eMBmDRpEsXFxUfzdCilHO707PHP/E1E\nV1da1URdcwvDk7u+nvENN9zAwoUL2bdvHzfddBMvvPACFRUVrFq1itjYWAYOHIjH4znqx9+5cye/\n/e1vWbFiBZmZmdx+++3HtJ6D4uPjD/3tcrm01KOUOkLU9/h9/iC1TS1kJcfhjun66bjppptYsGAB\nCxcu5IYbbqC2tpZevXoRGxvL+++/z65du7pcfurUqbz44osAbNiwgXXr1gFQV1dHcnIy6enp7N+/\nnzfeeOPQMqmpqdTX17db1wUXXMBrr71GU1MTjY2NLFq0iAsuuOBoN18pFYVOzx5/BFU22B+YzkmJ\n72ZOGDVqFPX19eTn59OnTx9uueUWrrrqKsaMGUNhYSHDhw/vcvl77rmHOXPmMGLECEaMGMGkSZMA\nGDduHBMmTGD48OH069eP884779Ayc+fOZcaMGfTt25f333//0PSJEydy++23M3nyZAC++c1vMmHC\nBC3rKKW6FdWXZfYHg2zZW09aYiz9o/D3UPWyzEo5x9FcljmqSz1VjT6CxpAbRm9fKaWcIqqDv7a5\nhaQ4N4lxeuVNpVT0OK2CP5JlKX8giMcXIDUhOk9znIolPqXUyXHaBH9CQgIHDhyIWGA1+gIYIDk+\n+oLfGMOBAwdISEjo6aYopXrAaZN6BQUFlJSUUFFREZH11TS10OTz4647/guynY4SEhIoKCjo6WYo\npXpAWMEvIjOAxwAX8LQx5jdt7s8E5gNnAB7gDmPMhtB9xUA9EAD84Z51bis2NpZBgwYdy6Iduuy/\nl9MrLZ7n75zY/cxKKeUg3ZZ6RMQFPAHMBEYCN4vIyDaz/RRYY4wZC3wDu5NobZoxZvyxhn6kVdR7\n2bq/nilnZPd0U5RS6qQLp8Y/GdhujCkyxviABcA1beYZCbwHYIzZAgwUkbyItjSCPi06AMC5Z+R0\nM6dSSjlPOMGfD+xpdbskNK21tcB1ACIyGRgAHCwgG+AfIrJKROYeX3Mj4+MdB0iNdzO6b1pPN0Up\npU66SJ3c/Q3wmIisAdYDX2Br+gDnG2NKRaQX8I6IbDHGLG+7gtBOYS5A//79I9Ssjn28o5KzB2fh\ndp02g5qUUipiwkm+UqBfq9sFoWmHGGPqjDFzjDHjsTX+XKAodF9p6P9yYBG2dNSOMeZJY0yhMaYw\nNzf3qDckXCXVTew60MQULfMopaJUOMG/AhgiIoNEJA6YDSxpPYOIZITuA/gmsNwYUyciySKSGpon\nGbgU2BC55h+9T3YcrO/riV2lVHTqttRjjPGLyH3AW9jhnPONMRtF5O7Q/fOAEcCzImKAjcCdocXz\ngEWhcfJu4EVjzJuR34zwfbLjAFnJcQzLS+3JZiilVI8Jq8ZvjFkKLG0zbV6rvz8BhnawXBEw7jjb\nGFGb9tYxvl9Gp7+0pZRSThdVZzeDQcPOykYG5yT3dFOUUqrHRFXw763z4PUHGZSrwa+Uil5RFfw7\nKxoBGKQ9fqVUFIuq4C+qbADgjNyUHm6JUkr1nOgK/opGkuJc9ErVX9xSSkWvqAr+nZWNDMpJjsrL\nMCul1EFRF/yDtcyjlIpyURP8Xn+AkuomPbGrlIp6URP8uw80ETToGH6lVNSLmuAvqtShnEopBVEU\n/DsPBr9+eUspFeWiJviLKhrISYknLSG2p5uilFI9KmqCX6/Ro5RSVlQFv9b3lVIqSoK/trmFygYf\ng7W+r5RS0RH8O3VEj1JKHRIlwW8vzqY9fqWUipbgr2gkRqBfVlJPN0UppXpcVAR/UWUj/bKSiHe7\neropSinV46Ii+IsPNDIwW8s8SikFURL8e2s89M1I7OlmKKXUKcHxwe/1BzjQ6KN3WkJPN0UppU4J\njg/+8jovAL3T9Ve3lFIKoiD499V5AOidrqUepZSCaAj+2lDwa6lHKaWAMINfRGaIyFYR2S4iD3Zw\nf6aILBKRdSLyuYiMDnfZE21/nQa/Ukq11m3wi4gLeAKYCYwEbhaRkW1m+ymwxhgzFvgG8NhRLHtC\n7a31kBjrIi3RfTIfVimlTlnh9PgnA9uNMUXGGB+wALimzTwjgfcAjDFbgIEikhfmsifUvjoPvdMT\nEJGT+bBKKXXKCif484E9rW6XhKa1tha4DkBEJgMDgIIwlyW03FwRWSkiKysqKsJrfRj213rIS9MR\nPUopdVCkTu7+BsgQkTXAd4AvgMDRrMAY86QxptAYU5ibmxuhZtlSj9b3lVLqsHAK36VAv1a3C0LT\nDjHG1AFzAMTWVHYCRUBid8ueSMGgobzeo0M5lVKqlXB6/CuAISIySETigNnAktYziEhG6D6AbwLL\nQzuDbpc9kaqafLQEDL211KOUUod02+M3xvhF5D7gLcAFzDfGbBSRu0P3zwNGAM+KiAE2And2teyJ\n2ZT2Do3hT9dSj1JKHRTWGEdjzFJgaZtp81r9/QkwNNxlT5bDwa+lHqWUOsjR39zdp1/eUkqpdhwd\n/PvrPMQI5KTEdT+zUkpFCUcH/95aD7mp8bhdjt5MpZQ6Ko5OxP11OpRTKaXacnTw76v1tB/KGWiB\nYLBnGqSUUqeAKAj+Nid2F9wC886DhshdFkKdJMb0dAuUcgTHBn+j10+9139kqad0FXz5FpRvgr/M\nguaanmugOjpNVfA/E+Hzp45t+WBAj/SUCnFs8B/+5a1WpZ6P/wfi0+GGZ6FiK7xwPXgbeqiFCmPg\nwA5YuwCWPwwN5Z3P+8VfoKoI3voZVGw7+sdaeAc8Psk+3unMU9vTLVDH4sCOrt/fJ5ljL1K/r6YZ\ngLyDpZ6qnbBpMZz7HRh1LcS44OXb4OmLYdJtMGoWpPYO/wGaa2DXPyEuBVJ62WUTM8NfPuC3bYjW\ny0UX/xP+9zZobFVy+3QeXPM4DJt55LzBAKx4CvqMh5rd8NrdcMfb4Arz7bvrY9j0GkgMzJ8Bt74C\nfcZGbltOhoAfXrnDvoezzoDBF8KZl8DQGRDj2P5bzwi0wP6NULICyjdDfKr9jKfkwcDzjy4nSlfD\nh/8FW/4GqX3gG0sgt8Pvup5UYk7BumlhYaFZuXLlca2j4nfT2Vzhpf+3/srAgnxY+iNY+Qx8dz2k\n9bEzbfk7fPAb2LfOhsLwK2HWPIhL7nilwQDs+QxWPWuDxO85fJ/EwIz/C2fP7b5xpavhpZuhzzi4\n4ZmuH89bD4kZ4W10cw3sXAZDLoXYU3g0k68J/jDF9vgv+D4UTAYMLPoW7FsPE2+Dmf8PYkM77a1v\nwEuz7ZGaCdje+8W/gPO/1/1jGWPDvroYvvZXWPA1+5xePx/O+Ird+UZC5Xb4/I8waQ7kRfi3hoJB\nu7Nb91f73NTvs50OXwP0nQgzfgP9z+5+PdXFkDGg685GSzO44qN3Z7LtLfv+8oUqAfHp0NIEwZbD\n8+RPgiGXQXwK+Brtc5acC5kDIL0f1O6xO41dn8CeTyEh3b5uaxeACcI3FkPv0I8UVu+CoB+yzzju\npovIKmNMYVjzOjL4A36Cv+xFDAGCOcOI+epT9sM/ahZc+/v281dsg7UvwT8ftSF0y8v2xQJ7iLbm\nBftClq62b4i4VBh7A4y+3r6QjRX2Rf3yLbj8tzD5rs7btvVNWDjH9iIaKyC/0AZSUtaR8x3YYXcO\nlVttT6HXSDhzOpx9d8dhVb0LXrjBzp+SB1Pug8I77JvzRNn2FjTsh/G3Hl1QvPOQfa5vex0GTT08\n3e+F9//D3jf6q/DVP9mQen6WLc09sC50pPYN2PYmzF3WfchufRNeugmu/G/7fNSW2PVVboOEDBh0\nAYy8FsZcH17bvfXw4SP2Q15wFqT0hg9/C58/aT/AGQPgW8sP76w9tfDat+32jL6u63U3V8O6l2H1\n8/bvkdfAmK/aMtfK+fCVn8PUH9l5Ay2wfiG8+29QvxcGT7OPX11sg+i6J+375aC1f4VFc+0O9exv\ndfz4NXvsEXDf8TD7xc53itW77GP0Gh7ec3a6qNwOT02zr+3537evb3qBvc9TY7d7+zu2I1K66vBy\n4rIdktZi3NB7DIy4Gs76JiSk2fU/d7XdWUz8Bmx/F8pDly7rd46tPIy8FuKSjqn5GvzVxfDYOF7j\nIq6N/8KGtQnAtz+FXiM6X27ja/DKnfYFu/px24P74gV7X+8x9o3Q/xxbimjbS/f7bOli61K44r/s\ni93Wij/B0h9C77HwtZft0cMrd0L2mXD9M5A7zAbdzuU23MAGfdVO2xMu3whnXgzXPXXkjqLsC3jx\nJnsEMv0h2LwEij6wO6/BF8GgC+0hanKubbcr7vhLTNXF8PsptjfU7xy45gnIObP75fathz9eCONu\nhmuf6Hiej/4b/vELuOgnNjAfLzwy9Boq4PfnQFI23PWu3Yl2JBiEeeeDvxnu/RxcsXa6t95+eIuW\n2eeprgQu+KF9jO6el+W/hfd+2Wai2A/y0Mvs6zZ0Btz0F/vcPD/Lvs6Zg+A7qzveQRoD7/0KPnnc\nvoZ9xtud/Y53IeCz85z3XXuU07Z9vkb452P2aCAlz+549q6FulKY84YtaZWugvkzIeCF7CFw34oO\n1tME8y+Dii32Mc+9Hy5ts52eWnsu5tN59vN07v1w0YOn9tFluLwN8PR02xmbuwwy+nU9/8FzLbHJ\ndgfZXG0/EzW7ILWvfd47el6qi+HZq+1RQf8pMOxy+1yufg4ObIfkXvC9DeA++isKa/AXLYPnruZH\nyb/i4dsuhhduhPwJcONz3S+79Q374Q34bEAW3gEX/MDW+Lrj94V6o2/YwJ72M7un9zbYwF/7kj1E\nvH7+4Z540TJbfvA12CDrM84Gf9YZ8LUFkDXYzmcMrHoGlv4Y0vPh0l9Dwz7Yv8kebSRlw60L7c4D\nYM8KO3/RBzYEWnMn2KOV878XXli3ZQz85as20Kb9DJb9xvbWJ82xj5850O6YWppDO10gORuScuzz\nU7Pbhk/bo5zW6198H6z5i30+yjfD9zZBSqsf6Nm5HJ67xpbnbnyu48Be85ItkVw/3+5AOhIMwN++\naz94U+6DS3/Vefj7vfDfoyFvlN25l6y0R1ijZtmOAcDHj8PbP4Pp/2pf2+IPYcwNNpi/sdjuiNtu\n61s/hU9/b1+T8+632ww2TDa/Di0eexQZ7s66rsz23IMB23P/6y0QE2vX8c7/gduXwsDzjmzDwjtg\n4yJ79LntLVj5J5j1JIy7yY6oWvOi3SE3HYAJt9jS5urn7I7k2j9Av7PCa5sxdsfUZ9zxdz4Cfihb\nHXqel0OnajtLAAATu0lEQVR8Gkz4uu0ctT3/U1tij5z2rYfkHNsJSsmz79XMgfD+r+35k68vav8a\nRVqgxe6wW5dwjbHnoio2d9xpDIMG/6pn4fX7+X7f53lk7tX2A2BM+CcDdy63b/6zvwUZ/Y/usf1e\n+0Fe8Sf7xrrgB/bE5IHtcOGDMPWH7Q+ha3bbw76SlVC60h6VXPXY4XJTa3tW2PCsL7O349Nsz+Hq\n/4HUvPbzHxw5U/I5eOpsENfstkEU8NlDy7E3woDz7E4qHOv+F1795uGyQf0+eOPH9pxJ0N/98l/9\nU/elFb8Pnr/W1rLH3mRLF219/D/w9s/h4n+D8797eHrNbnvuZu1L9ujqrve7LkUFg/Dmg/YIb8yN\ntueeOdAeibX+cK5+DpZ8B77+GpwxreN1GQN/vdWezAOY9Uf7HD8y3B553fjskfO+/XPb0z/7Hpjx\nn5E72b9/oy1veusgNgnufNt2Jv5rOAy9FL769OF5P3zEloymP2TPuQRa7E61ZCUMm2HLZQEvDLzA\n7hj7jrfL7XgfltxvyyDf23Dk+7VkFXzxPMz8v0f2Xj/7o32vjLgKrvl9x++5TYvh7z+w9fIx18Oo\n6w6fl2u9/oVzbA8bsTXz+v3QWG573GdMswMv4pJg3wbY/g/A2B2Vp8buwEyb4b0X/yK880anKA3+\nd/8d/4eP8pMR7/DwTZMi17CjUbIK/v4927tJ6W0/aIMuiMy6m6pg7xr7Jk4vOLawaCiHT56wOyhf\nva1T5k+064tLtmHha7SHvo0V9shjzA32ZOIfzrXBeOfbR+7EggHb26wutr3VuORQSUygqdKuxxUP\n42aH1+amKhtI597f8ckvY+yHf9NiGH+L3ek019gPucTYHu4FP+j8yKLtut79d9urJfSZcCfA7Bds\nDzIYhN+fbUPsWx923f7mGlvCG3mNLQEBvPlTu2P5/ubDR48Hz3VM/pYNyEiP8CpaBovutuseebWd\ntvRHsOrP8IOt9nnZ+qY9cT76usPnVAAaK+Gpr9iQHDvbbsfBE5Kt7V0Lf5x6ZGgaA3+8wPauL/ml\nPYoB+356bBy4E+1RaPYZcNMLh0e5BPzw3r/b0tXBo569awGxJ+In3QZDZ9rzKf94yAb8xQ/Z+5Ky\n7A5r25t2B71vA7Q02sdMzrXvj4lft+9bsO/VhnK746gutmXAUded1qPsNPgX3sGeDR8xb9wr/HrW\nmMg17GgFA7Z01P8ce3h5Kmrx2KOBomW2d91YYeu9LY22x5ScY8tIZWugucruIERsHbSjIDjZvA02\nuMo32XprXJI9Apr6I1sSO1q+RnvEUF1sD/8rt9vhn946+zjXPW1P7B+tiq3wxOTDAfnpPHjzX2wp\n8YpHTlzgGHPkuvdvtDvuS38NQy6Bp6ZD9mCY82b7k4q+JrsDPTi6qjPPXWuf/wfW2Xk3v26PelJ6\n2/Mc31lty3QHjyzueNseQfzvHPt85w61te2mA7Z0U3inPfpxx0Pll/aE95oX7M4iNsmuc/iVduhv\nOEOo2z4HDqXB/9R0Pinx8nbhH3noqlGRa1g0C7TYQ/uNi6Bg0jHXIU8rjZXwzEyo22t3It4GeGDN\n4ZPER+uZy+0R0SX/bst1w6+w5yciNaQ0XE9fYo/AELtDm/vB4dErxyJ0To0rH7XDFuedb4P9pr/Y\nvyd83e7wHhsH/SbDLf9rl6stsWPca0ts79vXaEt2E25t/xjBgD2S2/CqPZ9QeGdUhPnROJrgd+YX\nuGp2sduMIc4dpWORTwRXrK0ND720p1ty8iTn2BOy82fY0S6X/cexhz7ApNvh1bvs6K+CQjs662SH\nPkDhHHjtHnvC97bXjy/0wQ7J7TsBPv6drdmXb7RHRr1GwFl32RJXc7UtG33l54eXSy+ww2zDEeOy\n516GXnZ8bVWAEy/Z4G2Axgp2BXKJ1+vwq+OV1teG44X/YoP7eIy42o5syhgANy845vHax23ULHui\n+ZonYMCU41+fiB1uWlVkT/bmDDv8nYULf2xP+m56zZ7zOFi7Vz3KeT3+mt0A7DG5DI/tgd6Ucp7M\nATDtp8e/ntgEmPu+/d7B0VzeI9JiE+G2JZFd54ir7Kihqh12bP/BI5mkLDvq6u3/Y4f+qlOC84K/\nuhiA3aYXY7XHr041Rzs8+HQR47InZDcttsNXW5t0mx3JdQxfSlInhvOCv2YXAHtML63xK3UydVWD\n19A/pTgvGat3EYxNpopUDX6llOqA85Kxuhh/Wn9AiNNSj1JKtRNWMorIDBHZKiLbReTBDu5PF5HX\nRWStiGwUkTmt7isWkfUiskZEju9ay+Go2YUv1V5gSXv8SinVXrc1fhFxAU8AlwAlwAoRWWKM2dRq\ntnuBTcaYq0QkF9gqIi8YY0KXFmSaMaYy0o1vxxioLsbTyw5R0+BXSqn2wknGycB2Y0xRKMgXANe0\nmccAqSIiQApQBYRxta4Ia6yEliaaku0XUjT4lVKqvXCSMR/Y0+p2SWhaa48DI4AyYD3wgDGHLn1n\ngH+IyCoRCePnqY5DaERPU6INfv0Cl1JKtRepZLwMWAP0BcYDj4vIweutnm+MGQ/MBO4VkakdrUBE\n5orIShFZWVFR0dEs3QuN4a9Psvsl7fErpVR74SRjKdD652gKQtNamwO8aqztwE5gOIAxpjT0fzmw\nCFs6ascY86QxptAYU5ibm9vRLN0LBX9dQl9Ag18ppToSTjKuAIaIyCARiQNmA22/770bmA4gInnA\nMKBIRJJFJDU0PRm4FNgQqca3U10Myb3wYC8jq8GvlFLtdTuqxxjjF5H7gLcAFzDfGLNRRO4O3T8P\n+CXwZxFZDwjwL8aYShEZDCyy53xxAy8aY948Qdtia/yZA/AF7A8f6zh+pZRqL6xLNhhjlgJL20yb\n1+rvMmxvvu1yRcDJuxxf9S4oOAuf355X1h6/Ukq155xkDAbsNb8zB2rwK6VUF5xzkbYYFzy4GwI+\nvJ/Yc8/xLr0ss1JKteWsLrEIuOPxBbTHr5RSnXFkMmqpRymlOufIZPT5g7hjBFeM/hizUkq15djg\n196+Ukp1zJHp6Ato8CulVGccmY4+f1C/vKWUUp1wZDpqqUcppTrnyHT0aqlHKaU65ch01FKPUkp1\nzpHp6PUHidcev1JKdciR6ejzB7TUo5RSnXBkOurJXaWU6pwj09EX0Bq/Ukp1xpHpqD1+pZTqnCPT\n0Qa/XpJZKaU64tzg11KPUkp1yJHpqNfqUUqpzjkyHXUcv1JKdc6R6agnd5VSqnOOS0djDL6A9viV\nUqozjktHf9BgDHpyVymlOuG4dNTf21VKqa45Lh01+JVSqmthpaOIzBCRrSKyXUQe7OD+dBF5XUTW\nishGEZkT7rKR5gto8CulVFe6TUcRcQFPADOBkcDNIjKyzWz3ApuMMeOAi4D/EpG4MJeNqEM9fq3x\nK6VUh8JJx8nAdmNMkTHGBywArmkzjwFSRUSAFKAK8Ie5bER5tdSjlFJdCicd84E9rW6XhKa19jgw\nAigD1gMPGGOCYS4bUQd7/DqcUymlOhapdLwMWAP0BcYDj4tI2tGsQETmishKEVlZUVFxzA3RGr9S\nSnUtnHQsBfq1ul0QmtbaHOBVY20HdgLDw1wWAGPMk8aYQmNMYW5ubrjtb8fbEgAgzqVX51RKqY6E\nE/wrgCEiMkhE4oDZwJI28+wGpgOISB4wDCgKc9mI0h6/Ukp1zd3dDMYYv4jcB7wFuID5xpiNInJ3\n6P55wC+BP4vIekCAfzHGVAJ0tOyJ2RRLx/ErpVTXug1+AGPMUmBpm2nzWv1dBlwa7rInkg7nVEqp\nrjkuHbXUo5RSXXNcOnp1OKdSSnXJcemoNX6llOqa49JRa/xKKdU1x6XjwRp/fKzjNk0ppSLCcemo\nPX6llOqa49LR5w8SI+DW4FdKqQ45Lh19Af2hdaWU6orjEtLnD2qZRymluuC4hPT6g8S59QJtSinV\nGccFv88f1C9vKaVUFxyXkFrjV0qprjkuIX3+gNb4lVKqC45LSJ9fe/xKKdUVxyWkV4NfKaW65LiE\n1OGcSinVNcclpJ7cVUqprjkuIbXGr5RSXXNcQmrwK6VU1xyXkF5/kHit8SulVKccl5Ba41dKqa45\nLiH1kg1KKdU1xyWk1viVUqprjktILfUopVTXHJWQgaAhEDTEufSyzEop1Zmwgl9EZojIVhHZLiIP\ndnD/j0RkTejfBhEJiEhW6L5iEVkfum9lpDegtUO/t6s9fqWU6pS7uxlExAU8AVwClAArRGSJMWbT\nwXmMMQ8DD4fmvwr4njGmqtVqphljKiPa8g5o8CulVPfCScjJwHZjTJExxgcsAK7pYv6bgZci0bij\n5Q0EAA1+pZTqSjgJmQ/saXW7JDStHRFJAmYAr7SabIB/iMgqEZnb2YOIyFwRWSkiKysqKsJoVnsH\ne/z6BS6llOpcpBPyKuCfbco85xtjxgMzgXtFZGpHCxpjnjTGFBpjCnNzc4/pwbXUo5RS3QsnIUuB\nfq1uF4SmdWQ2bco8xpjS0P/lwCJs6eiE8AU0+JVSqjvhJOQKYIiIDBKROGy4L2k7k4ikAxcCi1tN\nSxaR1IN/A5cCGyLR8I4c6vFrqUcppTrV7ageY4xfRO4D3gJcwHxjzEYRuTt0/7zQrLOAt40xja0W\nzwMWicjBx3rRGPNmJDegNS31KKVU97oNfgBjzFJgaZtp89rc/jPw5zbTioBxx9XCo+DV4FdKqW45\nKiG1x6+UUt1zVEJ6tcavlFLdclRCHhzVo5dlVkqpzjkqIbXUo5RS3XNUQmrwK6VU9xyVkD6/vVZP\nvFsvy6yUUp1xVvDrN3eVUqpbjkpI/eauUkp1z1EJeTD4Y13Swy1RSqlTl6OC3xv6vd3QJSKUUkp1\nwFHB7/MH9Vr8SinVDUelpM8f1BO7SinVDUelpAa/Ukp1z1Ep6Qto8CulVHcclZI+f1CHciqlVDcc\nlZJa6lFKqe45KiW11KOUUt1zVEp6W7TUo5RS3XFUSnq1x6+UUt1yVEr6/EH9ERallOqGo1LS5w9o\nj18ppbrhqJT0BbTGr5RS3XFUStpSj/4Ii1JKdcVxwa+lHqWU6lpYKSkiM0Rkq4hsF5EHO7j/RyKy\nJvRvg4gERCQrnGUjSYNfKaW6121KiogLeAKYCYwEbhaRka3nMcY8bIwZb4wZD/wEWGaMqQpn2Ui6\nZGQeo/qmnajVK6WUI7jDmGcysN0YUwQgIguAa4BNncx/M/DSMS57XB6dPeFErFYppRwlnLpIPrCn\n1e2S0LR2RCQJmAG8crTLKqWUOjkiXRC/CvinMabqaBcUkbkislJEVlZUVES4WUoppQ4KJ/hLgX6t\nbheEpnVkNofLPEe1rDHmSWNMoTGmMDc3N4xmKaWUOhbhBP8KYIiIDBKROGy4L2k7k4ikAxcCi492\nWaWUUidPtyd3jTF+EbkPeAtwAfONMRtF5O7Q/fNCs84C3jbGNHa3bKQ3QimlVPjEGNPTbWinsLDQ\nrFy5sqeboZRSpw0RWWWMKQxnXv22k1JKRRkNfqWUijKnZKlHRCqAXce4eA5QGcHmnA6icZshOrc7\nGrcZonO7j3abBxhjwhoSeUoG//EQkZXh1rmcIhq3GaJzu6NxmyE6t/tEbrOWepRSKspo8CulVJRx\nYvA/2dMN6AHRuM0QndsdjdsM0bndJ2ybHVfjV0op1TUn9viVUkp1wTHBfzJ/6asniUg/EXlfRDaJ\nyEYReSA0PUtE3hGRL0P/Z/Z0WyNNRFwi8oWI/C10Oxq2OUNEForIFhHZLCJTnL7dIvK90Ht7g4i8\nJCIJTtxmEZkvIuUisqHVtE63U0R+Esq3rSJy2fE8tiOC/2T/0lcP8wM/MMaMBM4B7g1t64PAu8aY\nIcC7odtO8wCwudXtaNjmx4A3jTHDgXHY7XfsdotIPnA/UGiMGY29xtdsnLnNf8b+fklrHW5n6DM+\nGxgVWub3odw7Jo4Iflr90pcxxgcc/KUvxzHG7DXGrA79XY8Ngnzs9j4bmu1Z4NqeaeGJISIFwBXA\n060mO32b04GpwJ8AjDE+Y0wNDt9u7MUjE0XEDSQBZThwm40xy4G2v13S2XZeAywwxniNMTuB7djc\nOyZOCf6o/KUvERkITAA+A/KMMXtDd+0D8nqoWSfKo8CPgWCraU7f5kFABfBMqMT1tIgk4+DtNsaU\nAr8FdgN7gVpjzNs4eJvb6Gw7I5pxTgn+qCMiKdifuPyuMaau9X3GDtVyzHAtEbkSKDfGrOpsHqdt\nc4gbmAj8wRgzAWikTYnDadsdqmlfg93p9QWSReTW1vM4bZs7cyK30ynBfzS/EnbaE5FYbOi/YIx5\nNTR5v4j0Cd3fByjvqfadAOcBV4tIMbaM9xUR+QvO3mawvboSY8xnodsLsTsCJ2/3xcBOY0yFMaYF\neBU4F2dvc2udbWdEM84pwR81v/QlIoKt+W42xjzS6q4lwG2hv2/jyF9CO60ZY35ijCkwxgzEvrbv\nGWNuxcHbDGCM2QfsEZFhoUnTgU04e7t3A+eISFLovT4dex7LydvcWmfbuQSYLSLxIjIIGAJ8fsyP\nYoxxxD/gcmAbsAP4WU+35wRu5/nYw791wJrQv8uBbOwogC+BfwBZPd3WE7T9FwF/C/3t+G0GxgMr\nQ6/3a0Cm07cb+DdgC7ABeB6Id+I2Y3+ffC/Qgj26u7Or7QR+Fsq3rcDM43ls/eauUkpFGaeUepRS\nSoVJg18ppaKMBr9SSkUZDX6llIoyGvxKKRVlNPiVUirKaPArpVSU0eBXSqko8/8BByUg98mjs4kA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c5fb495d68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24192/25000 [============================>.] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.86628\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 3. Advanced CNN sentence classification model - 2\n",
    "- Structure more similar to that proposed in **Kim 2014**\n",
    "    - Three convoltion operations with different filter sizes are performed and their results are merged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import concatenate, Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "filter_sizes = [3, 4, 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convolution():\n",
    "    inn = Input(shape = (sequence_length, embedding_dimension, 1))\n",
    "    convolutions = []\n",
    "    # we conduct three convolutions & poolings then concatenate them.\n",
    "    for fs in filter_sizes:\n",
    "        conv = Conv2D(filters = 100, kernel_size = (fs, embedding_dimension), strides = 1, padding = \"valid\")(inn)\n",
    "        nonlinearity = Activation('relu')(conv)\n",
    "        maxpool = MaxPooling2D(pool_size = (sequence_length - fs + 1, 1), padding = \"valid\")(nonlinearity)\n",
    "        convolutions.append(maxpool)\n",
    "        \n",
    "    outt = concatenate(convolutions)\n",
    "    model = Model(inputs = inn, outputs = outt)\n",
    "        \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def imdb_cnn_3():\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Embedding(input_dim = 3000, output_dim = embedding_dimension, input_length = sequence_length))\n",
    "    model.add(Reshape((sequence_length, embedding_dimension, 1), input_shape = (sequence_length, embedding_dimension)))\n",
    "    \n",
    "    # call convolution method defined above\n",
    "    model.add(convolution())\n",
    "    \n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.3))\n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.3))\n",
    "    model.add(Dense(1))\n",
    "    model.add(Activation('sigmoid'))\n",
    "\n",
    "    adam = optimizers.Adam(lr = 0.001)\n",
    "\n",
    "    model.compile(loss='binary_crossentropy', optimizer=adam , metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = imdb_cnn_3()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, y_train, batch_size = 50, epochs = 100, validation_split = 0.2, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8leX9//HXJyd7D0ICCSMIsncAJ0JxgAu1glht1aq0\njta231qt7be2v85va621Uq227lVFEay4sCAOlCF7QwgQAklIyE7OvH5/XCdkkAVkwDmf5+PBg5x7\nnPu6znjf133d130fMcaglFIqeIR0dwGUUkp1LQ1+pZQKMhr8SikVZDT4lVIqyGjwK6VUkNHgV0qp\nIKPBr5RSQUaDXymlgowGv1JKBZnQ7i5Ac3r06GH69+/f3cVQSqnTxpo1aw4bY1Lbs+wpGfz9+/dn\n9erV3V0MpZQ6bYjI3vYuq109SikVZDT4lVIqyLQZ/CLytIgUisimFuaLiDwqIrtEZIOIjGswb7qI\nbPfPu78jC66UUurEtKeP/1ngMeD5FubPAAb5/00CHgcmiYgDmAdcBOQBq0RkkTFmy4kU1O12k5eX\nR21t7YmsrpqIjIwkMzOTsLCw7i6KUqqLtRn8xpjlItK/lUVmAs8be2P/L0QkUUR6Af2BXcaYHAAR\nedW/7AkFf15eHnFxcfTv3x8ROZGnUH7GGIqLi8nLyyMrK6u7i6OU6mId0cefAexv8DjPP62l6Sek\ntraWlJQUDf0OICKkpKTo0ZNSQeqUObkrInNFZLWIrC4qKmppmS4uVeDS11Kp4NUR4/gPAH0aPM70\nTwtrYXqzjDFPAk8CZGdn6+9BKnUa8foMhyudFFU4Ka91U1HrwRgYmZlA74TIYxoaRRVO1u0vZW9x\nFTdM6kdUuOOkty9ASEjnNGg8Xh9Oj4+YiGMjs9btJTLs5Mrf1Toi+BcBd/v78CcBZcaYgyJSBAwS\nkSxs4M8BvtEB2+sWpaWlvPzyy9x5553Htd6ll17Kyy+/TGJiYovL/OIXv2Dy5MlceOGFJ1tMdRK2\nHSrns13FjM5MYFRmIuGh9QfEHq+PUEfrB8iFFbWkxkY0ezRVXuvm3yv389yKXI5UuUiMDicxOowx\nfRK59bwsBqTGnlTZnR4vW/LLWbe/lJ2FlUSEhhAfGUZcZCgh/vL4jKHK6aWi1k2tx8ucCX0ZkZFw\nXNspr3Xz1PIcVu4podbjw+n2UlbjprDCidfXfHstNS6CwWlxuL0+ql1eiiud5JfVdzOW1bj5n4sH\nt7jNTQfK2FVYyeFKJ0eqXSRGhTMgNYasHjHsLqri3Y0HWbK1gIE9Y3nu2xOJi2x+wMLq3BKWbi/k\nwJEa8ktrOVLtIiIshMhQB+GhIfiMwecDt89HjctLpdNDtf9/l8cHQI/YCIakx3FGagwHSmvYnF/O\nwbJaZmdn8vtrRuE4wR2PMYaCcicHy2oY2zfphJ7jeEhbP7YuIq8AU4AeQAHwILY1jzHmCbGf8seA\n6UA1cIsxZrV/3UuBRwAH8LQx5rftKVR2drZpeuXu1q1bGTp0aLsr1tFyc3O5/PLL2bSp8ahWj8dD\naOgpeQF0mzrrNTXG4DM0+hIUlNfy4ZYCdhRUMKBHDEN6xTM0PZ6E6NZHFRVXOkmMDm/XF6qgvJbF\nGw/i8RpGZiYwIiMBnzGs31/Kun2luL0+RvdJZEyfRFJiI46uV17r5i8f7uD5FXuPhldUmIMz0+Mo\nr3FTVOGk0ukhNiKU5Jhw0uIjuGR4OleNzaBHbAQ7Cip4+IMdvLf5EFePzeChWaOPltfl8fGXJTt4\n/vNcqlxeJmUlM7x3AqU1LkqqXHy+uxi318dFQ9O45dwsJmUlN2q1Oj1enB4f8Q3CzBjD2xsO8uZX\neRyudFJc6eJwpRO315Y9MToMj9dQ6fQ0+zpFhzswBkIE/nnTBM4+I+XoPI/XR4jIMS1nt9fHKyv3\n8ciSnZRUuRjXN5HYyDCiwkKIjQijV0Ik6QmRpMZFEB8ZRnxUKG6vYUOefe13F1USGeYgJiKU+MhQ\nhvdOYEzfRJ75bA9LtxXx8b1T6Bkf2WibtW4vv1u8ledX1F+Q6giRY3Yw8ZGhnDeoBx9sLmB0n0Se\n//bEoy1zYwxLtxfy+LLdrMo9giNESI+PJCMpiuTocFxeH7VuLy6Pj5AQwSFCqEOIDrdlrfs/JjyU\nUIewp6iKbYcq2F1USa+ESEZkJBDmCGH+mjxmjunNn2eNJtQRgjGGr/YdoaDcebScA3vGcmZa3NHH\nVU4Pr63ez9LtRWw+UEZxlYuUmHBW//zCE+qKFZE1xpjsdi3bVvB3h1Mx+OfMmcPChQsZPHgwYWFh\nREZGkpSUxLZt29ixYwdXXXUV+/fvp7a2lnvuuYe5c+cC9befqKysZMaMGZx33nl8/vnnZGRksHDh\nQqKiorj55pu5/PLLufbaa+nfvz833XQTb7/9Nm63m9dff50hQ4ZQVFTEN77xDfLz8zn77LP58MMP\nWbNmDT169Gi13G6vj6IK21IyBkRAsB/swv05xPfqx5k94xp90XOKKnlr7QH2FFdTXOnkSLWbc85I\n4d5LBjd7SOvx+nhn40GW7zjM7qJKdhdVUuX0HP2CuTw+1ueVATZQa9xewJZldGYiUwanMnVwT0Zl\nJhz9wJfXuvn94q28snI/qXERzBzdm6vGZjAoLZaIUFuGapeH7Ycq2JxfzvubD/HZrsM0zIS6707d\nRzxEODq/Z1wEKbER9IgNZ+vBCoqrnNwwqS+3nz+ArQcr+CKnmB0FFSTFhNMzLoKEqDDKatyUVLnI\nKapi44EyQkOEUZkJrN1fSkx4KOcOTOH9zQVcNaY3f549htJqF3e8+BUrc0u4YnRv5p4/gJGZjVvY\nRRVOXliRy/Nf7KW02k2f5CiuGZtJRlIU/91ayCc7i6j1+Jg6uCezszNJignnd4u3snZfKf1Tosnq\nEUNKbAQ94yIYmWHDtFdCFGC7P6pcHoyv7gWxoR/mCOFQWS3f/NeX7Cup5vEbxzEwNY7nVuTy2qr9\nOBzCpKxkzhqQgsvjY1VuCatyj1BW4+asAcn87NJhx9TjROUeruLChz9m9oQ+/O7qkUenb84v455X\n17GrsJJbz8viG5P60iMmgvioUMpq3OwuqmJ3USVp8ZGcPSCF8NAQ3t14kLtfWcv4fkn86srhvLfp\nEG+tO8De4moyEqO4/fwsZk/oQ3R4xzfU/r5sF398bzszRqQzrm8Sr6zaR05R1THLDe0VzzVjMyit\ncfHiF/soq3EzOC3ONlR6xzM8I4Hsfkka/HUaBv+v3t7MlvzyDt3msN7xPHjF8BbnN2zxL1u2jMsu\nu4xNmzYdHQ5ZUlJCcnIyNTU1TJgwgY8//piUlJRGwT9w4EBWr17NmDFjmD17NldeeSU33njjMcH/\nox/9iFvm3sGjf5vHxvXreOyJf/Dzn/yIvpkZ/Oje+3hn8btce9UVbN2zn6TkHviMwes1eHwGA4SG\n2FaL12corXZjDCREhR7tqvD5DFUuL/tydnL7ooMkRIUxrm8iIzIS+CKn+GjLqG9yNMkx4USGhfDZ\nrmIGp8Xxt2+MPdpqqXR6WPBVHv9YnkPekRp6xEZwZlosZ6TGEh8VysHSWvJKa/D6DFMHp3Lx8HQG\n9YylsMLJ1oO2W2LZ9iLW55ViDPRKiOSS4ekMTo/jkSU7KKpwcsOkfhSU17J0e+HRFm14aAixEaFH\nd2YAfZKjuHpMBleNzSA+KoyNeWVsyCsjRGBM30RGZSYS5hA25pWxdn8pOUWVlFS5OFzpIi4ylJ9c\nMuS4wmxHQQWvr97Psu1FTBuaxncmDyApJpx5S3fxp/e3c9GwNLbkl3O40skfrx3FzDGtD2ircXl5\nb/NB3lhzgM92H8YYSIuPYNrQNGIjQlmw9gBFFbb1mBoXwb2XDObr4zJPuGsB4EiVi5ueWcnm/HJ8\nxuAQYcbIXkSGhrAip5i8IzUAZPWIYWL/ZKaPSGfK4NQOHxjw4MJNvPjlPj744WQG9Ijhuc9z+d3i\nbSRGh/Hn2aM5f1C77jsGwKL1+fzg1bX4/I2cc85I4drxmVw+qjdhbXTVnax/fpLDb97ZCsD4fklc\nP7EvI/1daT5jWLmnhDfXHmD9/lJE4JJh6cy9YADjOqhr53iC//TsozgFTJw4sdEY+EcffZQFCxYA\nsH//fnbu3ElKSgoGe0hX4/LQPyuLrMHDOVRWQ9aQkWzYupMq/+G4MYbKWg9en2HkuRez53AVA4aO\n5D+L3iK/tIZlHy/n4adeZEdBBYPGn0d8QiIF5U5cobWEiOAIsf9CBJxug9tnwEBSdBipcRFENNNS\ndxdH8qdrR7Fm7xHW7D3C0u1FDOgRw/0zhnDN2IxGh97Lthfy49fXc8XfPiW7fxJ7iqqO9tOO65vI\ng1cMZ9qQnu06uZYWH0lafCRTBvfkBxeeSXGlk2Xbi3h30yFeXrkPl8fHkPQ4nvpWNqMy7bmR0moX\nS7YWUlBeS3mNmwqnh7S4SIb0imNoejx9kqMaBdLUIT2ZOqTnMdueNCCFSQNSjpl+vM5Mi+Nnlw3j\nZ5c1nn7X1IGIwB/f2056fCSvf/fso3VoTVS4g6vHZnL12EzyS2soq3EzJD3uaJ1+cslglm0vIr+s\nhq+Py2z2JOPxSooJ56XbJvHLRVvolRDJjWf1Iz2h/j0/UFpDmEPoGRfZyrOcvO9NG8T8NXn89p2t\nhDmE9zcXMG1IT/40azTJMeHH9VxXju5NdJiD3OIqLh/Vu1F9Ottt5w9gWO94fwMo7pj5Q3vFc9M5\n/dlbXIUjRMhMiu6ysjV1WgZ/ay3zrhITE3P072XLlrFkyRJWrFhBdHQ0U6ZMoaamhqKKWjxeH7nF\nVVRXVUNIKHuLqxAEJITKGie7iyopq3GTd6SGnMOV+IwhMS6Kfikx1KbFE+GAwelxhDlC6BEbQd/k\naMJDQ3CECEPS4+mZmtBsC8wY2/oPaaV15ggRZmX3YVa2HXxV4/ISGRbS7PNNGdyTd++ZzC/f3sz+\nkmrOGpDCGT1jmZSVzPgTPDStkxIbwdfHZ/L18ZlUOj1sPVjO6CYnVxOjw7l2fOYJb6Mr3TllIGP6\nJHJmWhw9GpxLaK/eiVH0ToxqNC3UEcKFw9I6qohHxUXaVnVzMpqUobP0iI3g9skDeGTJTkJDhJ9f\nNpRbz8s64c9UZ7xO7XXOGa13vQL0S4lpc5nOdloGf3eIi4ujoqKi2XllZWUkJSURHR3Ntm3b+OKL\nLyiucnGwrBYRoV9yNBXhPsIcIQzsGUtkqMO2wnHRJykahwiRYSH0TY4mzBFCn+QYEqLCjoZ2RKiD\nyeefx9J3FzLxvvv44IMPOHLEdse09OUQEY73a9PWkLrUuAjmfWNcq8ucrNiIUCb0T+7UbXSF9gSA\nqnf7+QMor/FwxeheXTKqJdhp8LdTSkoK5557LiNGjCAyKorklFRyD9uTN0MnnE/F3+Yx6MwhnDFo\nEGPGT6Csxk2P2AhCQ4TYyDDwhOEIkUYnlkJESIoJJzYylNS4SBKjWz6sffDBB7n++ut54YUXOPvs\ns0lPTycu7tjDSaVORzERofziimHdXYygcVqe3O0uTreXQ+W1lNd6MMYQERpCiAjGgA87Btjn72JJ\ni49ocUz3CW3b6cThcBAaGsqKFSu44447WLdu3Uk956nwmiqlOoae3O0E5TVu9h+pBiAlJpyk6DAi\nwxwt9q939MiHffv2MXv2bHw+H+Hh4Tz11FMd+vxKqeChwd8GYwyFFU4KymuJCnPQLyWa8NDW+8I7\n4z44gwYNYu3atR3+vEqp4KPB3wqfMeQdqaG02kVSdDgZiVGddi8QpZTqKhr8LfD5DHtLqqmoddtL\n0Tuwv14ppbqTBn8zPD4few9XU+3ykJEURUrM8Y/FVkqpU5UGfzMOldZS7fbSNzmahFaGWCql1Ono\nlPkhllOFy+PlSLWblJjwkwr92Fh7m938/HyuvfbaZpeZMmUKTYetNvXII49QXV199PGll15KaWnp\nCZdLKaU0+JsoqnCBcEKX2jend+/ezJ8//4TXbxr8ixcvbvXe/kop1RYN/gbcXh8l1S6SosMa3ScG\n4P7772fevHlHH//yl7/kN7/5DdOmTWPcuHGMHDmShQsXHvOcubm5jBgxAoCamhrmzJnD0KFDufrq\nq6mpqTm63B133EF2djbDhw/nwQcfBOyN3/Lz85k6dSpTp04F7G2eDx8+DMDDDz/MiBEjGDFiBI88\n8sjR7Q0dOpTbb7+d4cOHc/HFFzfajlJKnZ59/O/eD4c2duxzpo+k6OwHwdh70jR13XXX8YMf/IC7\n7roLgNdee43333+f73//+8THx3P48GHOOussrrzyyhZH/zz++ONER0ezdetWNmzYwLhx9fe9+e1v\nf0tycjJer5dp06axYcMGvv/97/Pwww+zdOnSY+67v2bNGp555hm+/PJLjDFMmjSJCy64gKSkJHbu\n3Mkrr7zCU089xezZs3njjTe48cYbO/DFUkqdzrTF7+czhpIqF4nRYUd/6KOhsWPHUlhYSH5+PuvX\nrycpKYn09HQeeOABRo0axYUXXsiBAwcoKChocRvLly8/GsCjRo1i1KhRR+e99tprjBs3jrFjx7J5\n82a2bNnSank//fRTrr76amJiYoiNjeWaa67hk08+ASArK4sxY8YAMH78eHJzc4/35VBKBbDTs8U/\n4w8d/pSF5bX4ymubbe3XmTVrFvPnz+fQoUNcd911vPTSSxQVFbFmzRrCwsLo378/tbW1La7fkj17\n9vDQQw+xatUqkpKSuPnmm0/oeepERNTXweFwaFePUqoRbfFjb8twpMpFXGRYsz8tWOe6667j1Vdf\nZf78+cyaNYuysjJ69uxJWFgYS5cuZe/evS2uCzB58mRefvllADZt2sSGDRsAKC8vJyYmhoSEBAoK\nCnj33XePrtPS7aDPP/983nrrLaqrq6mqqmLBggWcf/75J1J9pVSQOT1b/B2syunB7fXRq41f6xk+\nfDgVFRVkZGTQq1cvbrjhBq644gpGjhxJdnY2Q4YMaXX9O+64g1tuuYWhQ4cydOhQxo8fD8Do0aMZ\nO3YsQ4YMoU+fPpx77rlH15k7dy7Tp0+nd+/eLF269Oj0cePGcfPNNzNx4kQAbrvtNsaOHavdOkqp\nNultmYH9JdWU17gZ2is+qO7Fo7dlVipwHM9tmYO+q8fnM5TVuO0vXgVR6CulglfQB395rdv+zu1x\n/qizUkqdrk6r4O+Mbqkj1W7CHSHEtPF7s4HmVOziU0p1jdMm+CMjIykuLu7QwHJ7fVTWukmMDg+q\nWy4bYyguLiYysvWT2UqpwHTajOrJzMwkLy+PoqKiDnvOilo3ZTUeiI/gSP5psw/sEJGRkWRmZnZ3\nMZRS3eC0Cf6wsDCysrI69Dln/2MF1S4P//neuLYXVkqpABFczdwGnB4v6/aXclZWSncXRSmlulS7\ngl9EpovIdhHZJSL3NzM/SUQWiMgGEVkpIiMazMsVkY0isk5EWr/5fBfadKAMl8dHdv/k7i6KUkp1\nqTa7ekTEAcwDLgLygFUissgY0/AuYg8A64wxV4vIEP/y0xrMn2qMOdyB5T5pK/ccAWBC/6RuLolS\nSnWt9rT4JwK7jDE5xhgX8Cows8kyw4D/AhhjtgH9RSStQ0vawVbnljAgNYaUDvrBFaWUOl20J/gz\ngP0NHuf5pzW0HrgGQEQmAv2AuiEjBlgiImtEZO7JFbdj+HyG1XuPMFG7eZRSQaijRvX8AfiriKwD\nNgJrAa9/3nnGmAMi0hP4UES2GWOWN30C/05hLkDfvn07qFjN21lYSVmNW/v3lVJBqT0t/gNAnwaP\nM/3TjjLGlBtjbjHGjAG+BaQCOf55B/z/FwILsF1HxzDGPGmMyTbGZKemph53RY7HytwSAG3xK6WC\nUnuCfxUwSESyRCQcmAMsariAiCT65wHcBiw3xpSLSIyIxPmXiQEuBjZ1XPFPzOrcEnrGRdAnOaq7\ni6KUUl2uza4eY4xHRO4G3gccwNPGmM0i8l3//CeAocBzImKAzcCt/tXTgAX+2yGEAi8bY97r+Goc\nn1V7SpiQlRxUt2lQSqk67erjN8YsBhY3mfZEg79XAGc2s14OMPoky9ihDpTWkF9Wy9x+OoxTKRWc\ngu7K3VV7bP/+hCzt31dKBafgC/7cEmIjQhmSHt/dRVFKqW4RdMG/8UAZo/sk4NBf21JKBamgCn6f\nz7CrsJJBPeO6uyhKKdVtgir4D5bXUu3yMrBnbHcXRSmluk1QBf+uwkoADX6lVFALyuAfpMGvlApi\nQRb8FSRFh+kdOZVSQS3Igr9Su3mUUkEvaILfGMNODX6llAqe4C+uclFa7WagDuVUSgW5oAl+HdGj\nlFKWBr9SSgWZoAr+mHAHvRMiu7soSinVrYIq+M/oGav34FdKBb2gCv6BqdrNo5RSQRH8FbVuDpXX\ncob27yulVHAEv96qQSml6gVV8OuIHqWUCpbgL6ok3BFC3+To7i6KUkp1u+AI/oJKsnrEEOoIiuoq\npVSrgiIJc4uryOoR093FUEqpU0JQBH9BuZN0vXBLKaWAIAj+SqeHSqeHtHgNfqWUgiAI/oLyWgDS\nE/THV5RSCoIh+Mts8GuLXymlrIAP/kPlGvxKKdVQwAd/QbkTgHQNfqWUAtoZ/CIyXUS2i8guEbm/\nmflJIrJARDaIyEoRGdHedTtbQXktcRGhxESEdvWmlVLqlNRm8IuIA5gHzACGAdeLyLAmiz0ArDPG\njAK+Bfz1ONbtVIfKaknToZxKKXVUe1r8E4FdxpgcY4wLeBWY2WSZYcB/AYwx24D+IpLWznU7VUFF\nLWnxOqJHKaXqtCf4M4D9DR7n+ac1tB64BkBEJgL9gMx2rot/vbkislpEVhcVFbWv9O1QUFarJ3aV\nUqqBjjq5+wcgUUTWAd8D1gLe43kCY8yTxphsY0x2ampqhxTK5zMUVjj1xK5SSjXQnjOeB4A+DR5n\n+qcdZYwpB24BEPvbhnuAHCCqrXU70+EqJx6f0Ra/Uko10J4W/ypgkIhkiUg4MAdY1HABEUn0zwO4\nDVju3xm0uW5nKvQP5dTgV0qpem22+I0xHhG5G3gfcABPG2M2i8h3/fOfAIYCz4mIATYDt7a2budU\n5ViHyupu16DBr5RSddo1uN0YsxhY3GTaEw3+XgGc2d51u0rdVbvax6+UUvUC+srdgvJaQgR6xIa3\nvbBSSgWJgA/+HrERLf/y1paF8MnDXVsopZTqZgF9H4NDbf0Ay+d/g7xV0Pcs6HdO1xVMKaW6UWC3\n+Mtq6RnXQvB7XHBwg/373fvAd1yXHSil1GkrsIO/orblH2Ap3AxeJwy5HA5tgHUvdW3hlFKqmwRs\nV0+t20tptduO6Pn8MYhJhdHX1S9wYI39/5LfQtVh+Oj/wbCZEJnQPQVW7edxwoe/gOpiSD4DUs6A\nzGxIHtDdJVPqtBCwwV/3k4v9w0ptSCT2gVGzQcQucOAriO4Bif1gxh/gyanw3k/hnO9ByiBwBOxL\nc3ozBv7zI1j3IsRnwsb5gLHzUgbCwIsg+9uQ2uzo4s4tV9E26HEmhDiaX6a6BEpyoPwAVBRAZQFk\nnQ8DpnRlSZUK3OCvu3hr1MH5YLxwJBeKd0GPQXaBA19Bxni7I+g9FibcCqv+abt8QqNsCzL7Fhh6\nJTjC4PBOOwJo6yLweexzhMfCbUsgOevkCuv12HK0FBiq3orHbOhP/gl87WfgrrVhmvsJ7PwAVj8N\nG16FmxdDWhfdAXz/SljyK9j7qW04XPyb+nleDyz4Duz+CGqOHLvuyqfge6shtmfXlFUpAjj4Cyqc\nROAiI+ffNuAPrLHB0GMQOCts62z4VfUrXPoQZN9q+/sPboDti2H+tyGuN/QaBTveh9BIGHktRCfb\nk8ErHoNNb8DkH594Qb1ueHKKbf0NvRKGX21HGJ1KO4Ht70JSFvQccvzr7lwCH/8fGJ99HN8LrnjU\nvobHa8f78MH/2tdpyk/ttLBIG/Bpw2DSd6BkDzwzA56fCd9+z3YDdQZjYN8K+OxR2PGu7Ursew6s\n+DuMmgPp/t8i+uwR2DS/flryGfboMzbddlU9cR588HO45snOKWdnK9xmv0+Z2fVH0y0xxja4dn8E\n5fn2M++qgpmPQWLf49uuqxpW/sM22gZMaX6Z2jLbWKspgUt+DxGxx7eNACbGmO4uwzGys7PN6tWr\nT+o5nlqew673/87/hT0FN78D7/wPxKXDtxbCnk/gucvhhvkw6KLmn8DnhZ0fwpePw6FNMO6bcNZd\nENvgzqH/vNAG93c+bv45akrh72dDnwkw/f9s6DX15ZPw7r2QdYEdWuqutn3VU38Gw6+BkDbOv1cW\nwqY3ITQCIuLAEW6fw1lh5/c7B3oOs1/KigJY8wxsWQRTfwpDr2j9ub1uO+Jp9b9s8N/5hQ3a9vJ5\nYd5EW5a0EYCB3E8hcwJ8c4Et8zHr+GDzm7ZsDefXlMIjIyGpvw308JiWt1u4DZ69FMKiYdZz9nUB\n+39cetsB1Vw9SnKgthxclfbocdU/bSMhMhHOuRsm3QFeFzyWbbucbnkPirbCPy6AoZfDrGebf+6P\nfg2fPAQ3vQ1Zk204bnkLyg7AxNubf406U+E28NRCz6F22z4fHFpvd+A1JfbIJDYdSvfB5gW2jgA9\nh9sd76jZEBbV+DlL99kjm81vQdk+Oy06xT7P4R0w/ia47M/tL+POJfDOj6B0r+3u+/5aCG1wkabP\nC2tfsK9tdbF9v1OHwvUv289P+UFY/kfYsxwS+tjGQVKW/WzEptmdUFK/xtvc9CZ8+rB9bwddDAMv\nPOWO0kRkjTEmu13LBmrw//rtzVy3+joGpScg3/0UPvxf+OIJuC/XfmmXPAj35kBMyolv5LO/2vMH\nP9jYfItl3cvw1h0QEmq7jy580PY/17Xma47Ao+NsS/Bbi2xgb3/XtlIKN0P6SJjxx9avMXjnx7Dq\nqdbLGZ8BacNh91Lwue0XrqoIrv4HjJrV/DpVh+G1m2z3xdArbRfX134Ok+9t32sDsOF1ePM2G751\nR1d100bOgmueOjaEt74N/74RZvwJJs2tn772RVh4F9y6xO5I25K/Dp67ApzljaeHRdsd65DLYOoD\njedtnA/nvkLtAAAX7klEQVRv32OPCtNHQVwvOLAa9n0JrorGy6YO8QfddY13QnXlvPwRu5Mtz4c7\nv2z5c+augXmTbMh+8y147z77GoA9X3D5X6D/efXLu6rt/LUvQMEmW4+x34Q+k45/h9ZQzRFY8ktY\n86x9HBIGqYNtq7yqCBD72rmr/CuI/VwOv9qW/ct/2PJEJtrwH/tN+/p98mfbcDA+OONrdvnBMyAq\nyT7Nwrtg4xvwoy2NjwK9HijYCHtXwP4v7Q4XwFkJ+7+w5+FGzoJlv4OZf4exN9j5xsBr37Kf175n\nw/Q/2Lq9fjNIiN3+upfszuGMr0FVIRTngLOs8evR71z7/vY5C979id0Zpw6xz1VZYJfpOcxuo985\n9n1ousPrYhr8wF//+U/uyfsfmDkPxt5o9+7PXQFzXoYN/4aD6+Ge9SdX0OLd8Ldx9jDy7DuPnf/S\nLNul9M23YPGPYfd/YcBUmPWM/eC//zNYMQ++s9x2J9XxeW0X0n9/DdVH4O6VEN/72Of3+eDhIZCR\nDZc9ZL8UXqf9gobH2r9zltkuroPrYfClMHGubam8cr1tfV/5qG2B7/3MBlzZfvvBLs+3X6Ir/2ZH\nQ732LdjxAdy9ynZVNPXeA3Yb178CkfG2Dn8/y+70vvtZ4yOX5Q/ZutX10zf0+s22Jdl7LMxdVj/9\n+avgyB74/rr2B1zJnvrRW2C/tCU59oivcIttBETG189/9QbY+7nd4R5cD7WltqXY72z7GkWn2JCP\nSqo/imruPXlmhg0rDMx+AYZd2Xo5d3wAL8+yYStij/Z6DrOfmdK90O88O91VaT9zznLbQs0YZ7u/\nXJV2+W8usK3WhmXZNN+W5eB6e47ryr8de6S32f/5rC6Gs+60XaOHNsChjTbIB10EZ0yzR7vOSvv5\nCI+FuLT65zD+o7k1z9gdk9dl33tjbChfcB8kZB5b98Kt9nMy9edwgb9RUZIDz1wKFQft48S+tisN\nAIEzL4Fz77FHt0+cb49Q7vrSNqg2vwWv32S7Ai+4r/49Kt5tP/OHd9gd05Sf1p+bM8a+15WFUHEI\nDq6zjcPSfXZ7jjC7/DnftzuPgo22N2Dv5/b8jqvCNhTmvNz8d6OLaPADK38/ncHuLST8dIftnvC4\n4I8DYOTXYddH0GciXPv0yRf28XMhIh6+/W7j6TVH4E8D7Rfp4l/bD9dXz9kWemIfu7P49422xXjV\nvOafuyTHdhUNvBDmNHOdwd7Pbchc+zSM+PrxldtVbbe/+6P6abHp9lA2Ls0e8o6eA71G23ml++Gx\nCTYErnuh8XPt+ghevMb+3f98uPEN++V/41a49hkYcU3j5Y2BRd+zrdbb/guZ4+10Z6V9zcIi7et3\n55f2vEJlIfx5MJz3Q5j2i+OrZ3Pquvquf9W2PsHuqP44wIbizMdsGd3VrXcptaRgs+3iGTYTrv1X\n+9ZZeLcN5ssfqT+X4qqG5X+y71FYjO2jjutlg6vvOXZn6qy0O8rF90L/c233ZV3YLfsDLPs9hMfZ\nnVnlIbv83SvrW9x7lsNzV9r3+cpH69/vk1FdYo+eSnbDhNvqB1S05KVZkL8WfrAJMPCvi+zn7dKH\nbJ2aa/TU2fSGPRc3+3nb1//YRNuwuX3psSPzXNX26KVpN05zfF579L3nYxh/S8sDBbwee47nrTvt\njui6F+1rmPupbeg5K+xnKCLWdt3WnfvpBMcT/BhjTrl/48ePNyfF5zNVD/Y0y//yrcbTX73BmD/0\nM+bBeGM+f+zktlFn6e+NeTDBmIqCxtO/esFuJ29N4+l7VxjzxzPsvN+kG1OW3/rzf/KwXXbrf46d\n9869xvy6pzG1FSdWdnetMZ/PM+arF40p3m2Mz9f68h//0ZZl55L6abUVxjw8wpi/ZRuz+lk7/9/f\nNOaxifaf19v8c9WWG/O7Psb8u8F7tOF1u/6mN435ZZIxH/yvnf7FP+z0Q5tPrJ5NuWuN+XWaMYt/\nUj/twFq7jfWvdcw2juwzxuPumOdqjy+ftOVf+ZR9vOND+7l88zv170H+Ovu6LvyefVxVbMxDQ4x5\ndJwxzsquK2tTu5fZsq9+1pi37rR/b3+vfet6Pcb8dYwxT0w25u0fGvPLRGMOfNW55W1O4Xb7Ov4q\n2X4nH4y3n7E/DzPm931suR4aYoyrutOKAKw27czYwLxyt7KAaGopiW4yzHLQxfVD6nqP65htDb0C\nMLDtncbTNy+w1wj0Htt4et+zbBfGGdPssL/mTvg2dPbd9sToOz+2Jxfr+Hy2H3PghSc+WiE0wnZR\njb3B9nu31YVy9vfsqJRXb7B9uj6fvfCtbL/tQhh/k63TloW2i2vyvS2fnI6Is8Nlty6yJ0vBvmZx\nvWDoTPtebXitvtur57COG54ZGmHfh5wGJ+X3LLf/Z53fMdtI7NO114JMuM1+pt7/uT2X8+Zt9rzO\nZQ/Xvwe9RsNZd9gjz72f26OuqiL4+r9O7Mimo2RNtl0lH/zcniM5/8e2O6c9Qhy22+fgOnsuYdJ3\nj/3OdYXUM+G2j2D8zfYI4cY3bVfijzbD/fvgpv9ART588Xjj9Ta8bs+lffggfPU85H5mjzY7WWAG\nf/EuAMqimxzSDfSP4BFH4z71k9FzmA3NuhNyYA91c5bZE0nNhWlCJnzzTXvtQFscYXDFX21/50e/\nqp+et9JOG3ZVy+t2tLBIuPk/9mTjuz+Bf06DlU/a8wZ9z7LLnPM9mPIAnDnd1r81k75j34svHrdD\n73Z+YOsTEgJjrrf1W/uCPZnXtLvoZA2YYkekVByyj/cstydTG/aRn05E7Pms0Ah44Sq7w5z9PIRH\nN15u6gOQ0BdengPb/mO7znqP6Z4y1xGx/efOcju6relJ97aMvt42GBL62PMj3SUq0Y5OmvEHGDit\n8Qi4/ufac2yf/gWqiu20XR/Bgrm2W2jFPLsjfv2mkztJ306BOY7fH/wVMU2CP76XbVlAx7VwRGyr\nf8U8OxImpof9Qvk8bQdfe2Vm25baF3+3O5oJt9pWtSOi/S2jjhLfG2543bbM3n/Atmyb9rtPua/9\nzzVyFnz1gj068rrqz1WcOd2eWHzPP17/eM9htGXABfb/Pcvt+7RvhT2ncTqL72X76Rd8F676e/PX\nMITHwOUPw0vX2p3f2Xd3dSmbN+Iae6HloIuP/xqW0Ag7xNcRcWqP1b/wl/ZE9vI/2kbP/G/bwQO3\nfWjLXp5nM6QLBGTwm8O7cJkwnNHNdKPMerbj78Q54uv2Fs9/G2cPNXM/taMuOuJEWZ2Lfm1P9r7z\nP3Z0yZaFtlXRcFRKVxGx1zUMvhQwJ/dlO+duWP+yHV6b0Nfu5MB+mUdea0dXZIzv+PvwpI+yO5ac\nj+3Yblel7XI43Q2bCWfOaDyuvalBF9krm9NHtn2dSFcJcZzcjjepf4cVpdOkDoZx37Kf6V3+QRVz\nXqpvhCb177J6nCLvescyxbvINWmEhzWzX0s5o+Pv49JrtL11Q7/z7FWqez+z49Y78pDNEWpHyGRO\ngPm32Pu9dGU3T3NiUuwRzslIG277pr2uY1+zMf6x2SNnn9w2mhPisEGfs8yO3AD7/gWC1kK/Tv9z\nu6fREOym/NS27kt229F4J3u7lxMUmC3+4t3sMb0ID+3C/VrGeHtlYMFm2Pi6vZKzo4VHwzf+DU9P\nt+O7B0/v+G10h8k/9ne1XN94esY4uP2/kN6BR04NDbjAnlxe+yKkjTy5i/mUao+4dDvE1+u2R+zd\nJPCC3+cl5Mge9pjpRId2w/1u0obbf50lOhlufd+elAyUW0j3OwceyG/+CCljfOdtd8BU+/+RXHs7\nDqW6Qt21I90o8Lp6SvchPjc5Xd3i70pRSfZeKoGkC0YyHCN5gL3XCwRG/75S7RR4yVi8G4BcXzrh\nLf3IulJgdzYDptghpf3O7u7SKNVlAq+rp8QG/x7Ti4gwDX7Vhq/9zN7GI1C6zZRqh8AL/uJdeMPi\nOFwbry1+1bb43q3fC0apABR4yVi8i9qELEACt49fKaVOQuAlY/EuqmPtFbsR3TGqRymlTnGBFfzu\nWijdT1VsfwBt8SulVDPalYwiMl1EtovILhG5v5n5CSLytoisF5HNInJLg3m5IrJRRNaJyMndZL8t\nR3IBQ3lMXYtfg18ppZpq8+SuiDiAecBFQB6wSkQWGWO2NFjsLmCLMeYKEUkFtovIS8YYl3/+VGNM\n5999yH9zttKofoBbg18ppZrRnmScCOwyxuT4g/xVYGaTZQwQJyICxAIlgKdDS9oe/uA/Eml//ky7\nepRS6ljtScYMYH+Dx3n+aQ09BgwF8oGNwD3GGJ9/ngGWiMgaEZlLC0RkroisFpHVRUVF7a5AI8W7\nICaVKrF3u9PgV0qpY3VUMl4CrAN6A2OAx0Sk7tZ/5xljxgAzgLtEpNlr440xTxpjso0x2ampqc0t\n0raSHEgZiMtjb7uso3qUUupY7Qn+A0DDn47P9E9r6BbgTf9PP+4C9gBDAIwxB/z/FwILsF1HnaN4\nF6ScgctrDza0xa+UUsdqTzKuAgaJSJaIhANzgEVNltkHTAMQkTRgMJAjIjEiEuefHgNcDGzqqMI3\n4vXYnwTsdx5Otz/49cpdpZQ6RpujeowxHhG5G3gfcABPG2M2i8h3/fOfAH4NPCsiGwEB7jPGHBaR\nAcACe86XUOBlY8x7nVITR6j9YQPA9cF2RCDM0Q13fFRKqVNcu+7VY4xZDCxuMu2JBn/nY1vzTdfL\nATrpVzRa5vL4CHeEIN1xq1+llDrFBWRfiNPj0/59pZRqQUCmo9Pj0xE9SinVgoAMfpfHp1ftKqVU\nCwIyHZ0erwa/Ukq1ICDT0aV9/Eop1aKATEeXV4NfKaVaEpDp6HRrH79SSrUkINNRW/xKKdWygEzH\nugu4lFJKHSsg09GO6tFx/Eop1ZyADH4d1aOUUi0LyHTU4FdKqZYFZDo69cpdpZRqUUCmo7b4lVKq\nZQGZjk4dzqmUUi0KuHQ0xvhv0qajepRSqjkBF/x1v7erffxKKdW8gEtHl0d/b1cppVoTcOno9Ad/\nRFjAVU0ppTpEwKWjtviVUqp1AZeOR4Nf+/iVUqpZAZeOR7t6dFSPUko1K+CCX1v8SinVuoBLR5fX\nC+hwTqWUaknApaPTrS1+pZRqTcClo9Orwa+UUq0JuHR0efTKXaWUak3ApaNTg18ppVrVrnQUkeki\nsl1EdonI/c3MTxCRt0VkvYhsFpFb2rtuR6u/gEuHcyqlVHPaDH4RcQDzgBnAMOB6ERnWZLG7gC3G\nmNHAFODPIhLeznU7lEtv2aCUUq1qTzpOBHYZY3KMMS7gVWBmk2UMECciAsQCJYCnnet2KKfHDufU\nWzYopVTz2pOOGcD+Bo/z/NMaegwYCuQDG4F7jDG+dq4LgIjMFZHVIrK6qKioncU/ll7ApZRSreuo\ndLwEWAf0BsYAj4lI/PE8gTHmSWNMtjEmOzU19YQLoqN6lFKqde1JxwNAnwaPM/3TGroFeNNYu4A9\nwJB2rtuhnB4fIQKh2tWjlFLNak86rgIGiUiWiIQDc4BFTZbZB0wDEJE0YDCQ0851O5RLf29XKaVa\nFdrWAsYYj4jcDbwPOICnjTGbReS7/vlPAL8GnhWRjYAA9xljDgM0t27nVMXS39tVSqnWtRn8AMaY\nxcDiJtOeaPB3PnBxe9ftTE6PV1v8SinVioBLSKfHp0M5lVKqFQGXkC6PTy/eUkqpVgRcQmqLXyml\nWhdwCWlP7gZctZRSqsMEXELqqB6llGpdwAW/jupRSqnWBVxC6gVcSinVuoBLSO3jV0qp1gVcQjo9\n2uJXSqnWBFxCaotfKaVaF3AJ6dIWv1JKtSrgEtJewKXDOZVSqiUBF/x6ywallGpdQCWkMcYO59Rb\nNiilVIsCKiGd+nu7SinVpoBKSJdXf29XKaXaElAJqT+0rpRSbQuohNSuHqWUaltAJWR9i1+Hcyql\nVEsCMvi1xa+UUi0LqIR0erwAOpxTKaVaEVAJebSrRy/gUkqpFgVUQh7t6tEWv1JKtSigElJH9Sil\nVNsCKiGdOqpHKaXaFFDBX3flrrb4lVKqZQGVkE63HdWjV+4qpVTLAioh9V49SinVtnYlpIhMF5Ht\nIrJLRO5vZv69IrLO/2+TiHhFJNk/L1dENvrnre7oCjSkF3AppVTbQttaQEQcwDzgIiAPWCUii4wx\nW+qWMcb8CfiTf/krgB8aY0oaPM1UY8zhDi15M/TkrlJKta09TeOJwC5jTI4xxgW8CsxsZfnrgVc6\nonDHS1v8SinVtvYkZAawv8HjPP+0Y4hINDAdeKPBZAMsEZE1IjK3pY2IyFwRWS0iq4uKitpRrGO5\nPD4cIYIjRE5ofaWUCgYd3TS+AvisSTfPecaYMcAM4C4RmdzcisaYJ40x2caY7NTU1BPauNPj1RO7\nSinVhvak5AGgT4PHmf5pzZlDk24eY8wB//+FwAJs11GncHl82s2jlFJtaE9KrgIGiUiWiIRjw31R\n04VEJAG4AFjYYFqMiMTV/Q1cDGzqiII3R39oXSml2tbmqB5jjEdE7gbeBxzA08aYzSLyXf/8J/yL\nXg18YIyparB6GrBAROq29bIx5r2OrEBDTrdP78yplFJtaDP4AYwxi4HFTaY90eTxs8CzTablAKNP\nqoTHwaktfqWUalNApaTt49cx/Eop1ZqACn6nx6ejepRSqg0BlZIuj1dH9SilVBsCKiVd2uJXSqk2\nBVRKalePUkq1LaBSUi/gUkqptgVUSuoFXEop1baASkmn26e3ZFZKqTYEVPC7vNrVo5RSbQmolNQ+\nfqWUaltApeSFQ3syvHd8dxdDKaVOae26V8/p4pE5Y7u7CEopdcoLqBa/UkqptmnwK6VUkNHgV0qp\nIKPBr5RSQUaDXymlgowGv1JKBRkNfqWUCjIa/EopFWTEGNPdZTiGiBQBe09w9R7A4Q4szukgGOsM\nwVnvYKwzBGe9j7fO/Ywxqe1Z8JQM/pMhIquNMdndXY6uFIx1huCsdzDWGYKz3p1ZZ+3qUUqpIKPB\nr5RSQSYQg//J7i5ANwjGOkNw1jsY6wzBWe9Oq3PA9fErpZRqXSC2+JVSSrUiYIJfRKaLyHYR2SUi\n93d3eTqLiPQRkaUiskVENovIPf7pySLyoYjs9P+f1N1l7Wgi4hCRtSLyH//jYKhzoojMF5FtIrJV\nRM4O9HqLyA/9n+1NIvKKiEQGYp1F5GkRKRSRTQ2mtVhPEfmpP9+2i8glJ7PtgAh+EXEA84AZwDDg\nehEZ1r2l6jQe4H+MMcOAs4C7/HW9H/jIGDMI+Mj/ONDcA2xt8DgY6vxX4D1jzBBgNLb+AVtvEckA\nvg9kG2NGAA5gDoFZ52eB6U2mNVtP/3d8DjDcv87f/bl3QgIi+IGJwC5jTI4xxgW8Cszs5jJ1CmPM\nQWPMV/6/K7BBkIGt73P+xZ4DruqeEnYOEckELgP+2WByoNc5AZgM/AvAGOMyxpQS4PXG/jJglIiE\nAtFAPgFYZ2PMcqCkyeSW6jkTeNUY4zTG7AF2YXPvhARK8GcA+xs8zvNPC2gi0h8YC3wJpBljDvpn\nHQLSuqlYneUR4CeAr8G0QK9zFlAEPOPv4vqniMQQwPU2xhwAHgL2AQeBMmPMBwRwnZtoqZ4dmnGB\nEvxBR0RigTeAHxhjyhvOM3aoVsAM1xKRy4FCY8yalpYJtDr7hQLjgMeNMWOBKpp0cQRavf192jOx\nO73eQIyI3NhwmUCrc0s6s56BEvwHgD4NHmf6pwUkEQnDhv5Lxpg3/ZMLRKSXf34voLC7ytcJzgWu\nFJFcbDfe10TkRQK7zmBbdXnGmC/9j+djdwSBXO8LgT3GmCJjjBt4EziHwK5zQy3Vs0MzLlCCfxUw\nSESyRCQcexJkUTeXqVOIiGD7fLcaYx5uMGsRcJP/75uAhV1dts5ijPmpMSbTGNMf+97+1xhzIwFc\nZwBjzCFgv4gM9k+aBmwhsOu9DzhLRKL9n/Vp2PNYgVznhlqq5yJgjohEiEgWMAhYecJbMcYExD/g\nUmAHsBv4WXeXpxPreR728G8DsM7/71IgBTsKYCewBEju7rJ2Uv2nAP/x/x3wdQbGAKv97/dbQFKg\n1xv4FbAN2AS8AEQEYp2BV7DnMdzYo7tbW6sn8DN/vm0HZpzMtvXKXaWUCjKB0tWjlFKqnTT4lVIq\nyGjwK6VUkNHgV0qpIKPBr5RSQUaDXymlgowGv1JKBRkNfqWUCjL/HyFtdiCmbd6nAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c71225d710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24768/25000 [============================>.] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.87284\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 3. Advanced CNN sentence classification model - 3\n",
    "- Structure more similar to that proposed in **Kim 2014**\n",
    "    - More techniques are applied to generate more stable results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.layers import BatchNormalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "filter_sizes = [3, 4, 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convolution():\n",
    "    inn = Input(shape = (sequence_length, embedding_dimension, 1))\n",
    "    convolutions = []\n",
    "    # we conduct three convolutions & poolings then concatenate them.\n",
    "    for fs in filter_sizes:\n",
    "        conv = Conv2D(filters = 100, kernel_size = (fs, embedding_dimension), strides = 1, padding = \"valid\")(inn)\n",
    "        nonlinearity = Activation('relu')(conv)\n",
    "        maxpool = MaxPooling2D(pool_size = (sequence_length - fs + 1, 1), padding = \"valid\")(nonlinearity)\n",
    "        convolutions.append(maxpool)\n",
    "        \n",
    "    outt = concatenate(convolutions)\n",
    "    model = Model(inputs = inn, outputs = outt)\n",
    "        \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def imdb_cnn_4():\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Embedding(input_dim = 3000, output_dim = embedding_dimension, input_length = sequence_length))\n",
    "    model.add(Reshape((sequence_length, embedding_dimension, 1), input_shape = (sequence_length, embedding_dimension)))\n",
    "    model.add(Dropout(0.5))\n",
    "    # call convolution method defined above\n",
    "    model.add(convolution())\n",
    "    \n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(10))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(10))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(BatchNormalization())\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(1))\n",
    "    model.add(Activation('sigmoid'))\n",
    "\n",
    "    adam = optimizers.Adam(lr = 0.001)\n",
    "\n",
    "    model.compile(loss='binary_crossentropy', optimizer=adam , metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = imdb_cnn_4()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, y_train, batch_size = 50, epochs = 100, validation_split = 0.2, verbose = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4XMW9//H3bNOqd6valnHvxg0bgyk2YFNMx6aE8gsh\nIZCQe0lCyb2BBEgjyU2hBQgpJKaZYrqpxhBsg417N3KRLKtbXdo6vz9mZa1kSZZtSatdfV/P48fa\ns2fPmdnyObNz5swqrTVCCCEiiyXUBRBCCNH9JNyFECICSbgLIUQEknAXQogIJOEuhBARSMJdCCEi\nkIS7EEJEIAl3IYSIQBLuQggRgWyh2nFaWprOy8sL1e6FECIsrV27tlxrnX609UIW7nl5eaxZsyZU\nuxdCiLCklNrXlfWkW0YIISKQhLsQQkQgCXchhIhAIetzb4/H46GwsJCmpqZQFyViOJ1OcnNzsdvt\noS6KEKIX9alwLywsJD4+nry8PJRSoS5O2NNaU1FRQWFhIUOGDAl1cYQQvahPdcs0NTWRmpoqwd5N\nlFKkpqbKNyEh+qE+Fe6ABHs3k+dTiP6pT3XLCCHEsdBaU1DZyLbiGsZmJ5CbHNOlx31dVkeTx8eY\nrIRWDSC/X+P2+XHarR3ub0NhNSu/rmBgSjQjMuIZkhaL3dq1dnKD28sH20oZkhrL+NzELj3meEm4\nB6mqqmLx4sV897vfPabHnX/++SxevJikpKQO1/npT3/K7NmzmTt37okWU4iw4vdr8svr2F5cy47i\nWpo8PuaOzmBqXgpWS+tvlg1uLxsKqtlRXINSCrvVQozDyuwR6aTEOg6vtyq/gqc/zWfNvkNUNXgA\nsFoU88dl8q3TT2LiwKQjylBU3ciH20p55atCNhRWA5CTFM05YzLISnTy5d5DfLm3kupGjwnuAfGc\nlB5LalwUKTEOapo8LFlbyPbi2lbbtiiIddiIc9qId9o4KS2OkZnxjMiIJ8pmwac1DW4vH20v44Ot\nJTR6fNwwc3CPh7sK1Q9kT506Vbe9QnXbtm2MHj06JOUB2Lt3LxdeeCGbN29utdzr9WKzhe9xMNTP\nqzh+1Y0eGt0+0uIc2KwWPD4/W4tq+HJvJbVNXkZnxTM6y7RYvX4/Xp/GZlVE2dpvebbV5PGxpaia\nERnxxDtbRlRtLKzi5bWFOB1Whg+IZ2i6aZ3WubzUu7xoDTarwmG14HRYSXDaSXDaSIk15Wz22a5y\nHnxr6+FAtCiwWSy4fX7S46OYeVIqPr8Jv7I6F9sO1uLzH5lJdqti7ugMzho1gFe+KmRVfiXp8VHM\nGTWACblJDM+I4/2tJTy3ej+1Li/xThvpcVGkxDqoc3nZU16Py+sHYExWApdNziHBaee9rcWs2FWO\n2+tnSFos0/NSyEpysru0jl0ldeytaHkcwITcRBZOG8i5YzIpqWliV2kte8rqqWkyz0tVo4evS+vY\nU1FP22hNjrEzf3wWCyZmM62dA1tXKaXWaq2nHnU9CfcWixYtYunSpYwcORK73Y7T6SQ5OZnt27ez\nc+dOLrnkEgoKCmhqauKOO+7glltuAVqmUqirq2P+/PmcdtppfP755+Tk5LB06VKio6O58cYbufDC\nC7niiivIy8vjhhtu4I033sDj8fDSSy8xatQoysrKuOaaaygqKmLmzJm8//77rF27lrS0tBOqV6if\n1/6outGDUhBtt3b6lb3J42N3aR07S2rxa4h32oiLsrG1qIb3t5Wwdt8hfH6NRUFaXBS1TV4aPT4A\nlOKIAAFw2CycOSKdCyZkMTY7keU7Snl3czGbDlQzcWASs4amMSIjjo93lPLO5mJqm7w4rBZOHZbK\njJNSeX+r2a/TbsHn13h8Xc+IGIeVyYOSmZqXzKbCaj7cXsrAlGhuPWMYE3ITGTYgDp9f89H2Ut7e\ndJCNhdU47RZiHDaSYuxMzE1iyuBkxuYkYLOYg1lZrYvX1h3g1XUHqKh3kx4fxa1nDOWaUwYd0X1S\n5/Ly6roD7C6ppbzeTXmti7goG0PSYhmSHsvkQcmMzkpo9Zh6l5cGt4/0+Kgj6qO1psHto7LejdYw\nKLVr3T6Nbh/55XX4/eYbhd2qyDuG7pvOhH24/+yNLWwtqunWfY7JTuC+i8Z2eH9wy3358uVccMEF\nbN68+fAwwsrKSlJSUmhsbGTatGl88sknpKamtgr3YcOGsWbNGiZNmsRVV13FggULuO66644I9zvv\nvJPvfe97PPbYY3z11Vc8/fTT3H777eTk5HDPPffw7rvvMn/+fMrKyiTce5nWmtc3FPHR9lJqm7zU\nNnmIcdi4cEIW88dnERdla7Vuc59tk8fHsi3FLF69n9V7Kg+v47BaGJ2dwLTByUwcmERJTRNbimrY\nfKCa/PL6dluqAKMy45k7OoPMRCeltS5KqpuIdliZlpfC1LxkEpx2dpbUsu1gDcU1TditFhxWCweq\nGnln80FKalyHtzU6K4Epg5PYUFDN5qJqtIa4KBvnjc3krFHpbCioYtmWEvZXNjAoJYYbTs3jyqm5\nxNit7K9s4OuyerTWxEXZiI2yYVEKt8+Px+en0e2jpslDTZOX3SW1fLn3ENuKa4h12Lj97GHceGpe\nh33Yx8Lt9bO5qJoxWQndsr1w1dVwD9++hl4wffr0VuPD//SnP/Hqq68CUFBQwK5du0hNTW31mCFD\nhjBp0iQApkyZwt69e9vd9mWXXXZ4nVdeeQWAzz777PD2582bR3JycrfWJ9I1n+wqrm7C5fXR5PER\n47CRFhdFapyDgsoG1u47xFf7D2G1KEZnJjA6K4FRWfEMTY/Dabeyq6SW/3ltM6v3VJKREEVaXBTx\nTht7K+r50ZKN/O/SzUwZnMyheg/FNU0canDjtFmJcVhxef3UubwMSonhjjnDiXfaaPL4qG70sKGg\nmn+u2of7sz0AZCREMT4nkfnjMhmZmcDIzDgcVis1TR5qm7wMTInu0snBiQOTjuhfBvjphWNYs+8Q\nO0pqmT08jcGpsYfvq2pws7Okjgm5iYdD8sIJ2dx7/mgOVjeRkeBs1WVwUnocJ6XHHdNrUdPkwaoU\nsVHdFzEOm4XJg+Qz0VV9Ntw7a2H3ltjYlg/E8uXL+eCDD1i5ciUxMTGceeaZ7Y4fj4pq+WpntVpp\nbGxsd9vN61mtVrxebzeXvP9Zt/8Qv353O6vyKztdz2ZRjMlOQGtM2Ab6Uy0KBqfGUlDZQGyUjV9c\nOp5F0wZiCYSc1pqv9lcFTsZVkZnoZNKgJFJjHbi8fupdXpSC+eOymHlS6uHHBXN5fewsriMjMYoB\n8c7ufxKCWCyK6UNSmD4k5Yj7kmIc7S5XSpGdFN0t+09wyhXRodZnwz0U4uPjqa2tbfe+6upqkpOT\niYmJYfv27axatarb9z9r1ixefPFF7rrrLt577z0OHTrU7fvoq/x+zcYD1SzbUsyGgiosSmG1KKJs\nFrKToslNjiYjwXk4SOsC/+pdXvZVNPDJzjLS4hzcf9EYpg1JwWm3EmWzUO/yUVHnorzezYD4KCbm\nJhHtMK1Vr8/PnvJ6thfXsquklp0ldZw2LI075g4nLa51/6tSiimDk5ky+PhbjlE2a4+PkBCimYR7\nkNTUVGbNmsW4ceOIjo4mIyPj8H3z5s3jiSeeYPTo0YwcOZIZM2Z0+/7vu+8+rr76ap599llmzpxJ\nZmYm8fHx3b6fnrCpsJqtB6txe/24vH7GZCUwc2j7VxvvLKnlgTe3sjq/koRoO8kxdmqaPJTUuLBa\nFOOyE7BaFD4NjW4vn39dQZ3ryG83NosizmkjwWnnv+aO4ObTh3TQDdD+c2izWhieEc/wjPB4joU4\nFn32hGp/5HK5sFqt2Gw2Vq5cya233sr69etPeLs9/by+tKaAu17eSNvzgtPykvmvc0Yw86RU6lxe\niqub+Pfq/Ty7ah+xDiuXTc7F5fVxqN6D1ao4e+QA5oweQFKMo9V2tNZUN3oorXXhtFmJjbISG2Uj\nymaRK3BFvyMnVMPQ/v37ueqqq/D7/TgcDp566qmQlKPB7eX9rSVoDbFRZmheSqyDtDgHSTGOVifb\nnv40nwff2sbpw9P4xaXjiXZYsSrFmxuLeOTj3Vzz1Gqi7dbDw/csCq6ePog7zx3Z6qKUziilSIpx\nHBH6QoiOSbj3IcOHD2fdunU9su3gIXsdqXN5eXblPp76NJ/Kene761gtipykaIakxeK0W1i2pYTz\nx2fyfwsntbpw5hsz87hy6kBeWlPAnvIGMhKiyEx0Mi4nkaHHOPJCCHHsJNwjnMfnp6LOxaj/fZdJ\nA5M4dWgaY7MTONTgNmOna5oorm6ipNZFfmkdtS4vZ4xI59Yzh5IeH2VOXjZ5qWwwF4SU1bnYX9nI\nnvI6CiobuX7mYO67aGy7V9s57Va+MTOv9ysthJBwD3d+rXF7/eaCEq8frc14YIfNQoPby8HqJpq8\nfhZMzGZ7cS1/+HBnq6saE6PtZCY4GZAQxYUTs1g4bRCT2hk3LYQILxLuYcrn11TWuymvc+Hx+Ttc\nLzbKRkZ8FA9faa4bqG7wsLusjvS4KAYkRPXrK/2EiGQS7mHG4/NTWe+mos6N1+8nNspGZoLzcGtd\nAa5AS96izMUk28tb5rNIjLGf0FhtIUR46HM/1hFO4uLMicGioiKuuOKKdtc588wzaTvks60//OEP\nNDQ0HL59/vnnU1VV1WqdJo+PgsoGthfXUlLTRIzDytD0OIamx5Ec6yA2yobdasFmtRAbZSM5xkFi\ntEOGCgrRT0m4d4Ps7GyWLFly3I9vG+5vv/324bnhtdaU1jSxq6SO6kYPKTEORmTEk5cW263zdggh\nIouEe5C7776bRx999PDt+++/nwcffJA5c+YwefJkxo8fz9KlS4943N69exk3bhwAjY2NLFq0iNGj\nR3PppZe2mlvm1ltvZerUqYwdO5b77rsPMJORFRUVcdZZZ3HWWWcBZgrhsrIyXB4fP3ngV5w67WSu\nOGcmy154hpzkaIoPFDB69Gi+9a1vMXbsWM4999wO57ARQvRPfbfp987dULype7eZOR7m/6rDuxcu\nXMgPfvADbrvtNgBefPFFli1bxve//30SEhIoLy9nxowZLFiwoMPujscff5yYmBi2bdvGxo0bmTx5\n8uH7HnroIVJSUvD5fMyZM4f1GzZw0y238vBvf8e/XnmL5NQ09lXU4/VrdpTUUvjlFpYs/hfLP/2c\nxGgbM2bM4OyzziQ5OZldu3bx3HPP8dRTT3HVVVfx8ssvc91113Xv8yWECFvScg9y8sknU1paSlFR\nERs2bCA5OZnMzEzuvfdeJkyYwNy5czlw4AAlJSUdbmPFihWHQ3bChAlMmDDh8H0vvvgikydPZuKk\nSWzavIX3P1/LnvJ6/FpT7/LR4PbS5DEjXxKddr7etJYrr7iU3AHJxMfHc9lll/Hpp58CXZ9aWAjR\nP/XdlnsnLeyedOWVV7JkyRKKi4tZuHAh//73vykrK2Pt2rXY7Xby8vLaneq3M36/Zsv2Xfz6Nw/z\nwtsf44xL4Kf//V2sPg+DU82vs4zIjCctzfxCjM2iyEqKJs5pw1Xf/vG3q1MLCyH6J2m5t7Fw4UKe\nf/55lixZwpVXXkl1dTUDBgzAbrfz8ccfs2/fvk4fP3v2bBYvXgzAqjXr2LhxI/ll9WzdX4zDGU1y\nUhJR7lpWfvIhKXFRJEbbO5xq+PTTT+e1116joaGB+vp6Xn31VU4//fQeqbcQIrL03ZZ7iIwdO5ba\n2lpycnLIysri2muv5aKLLmL8+PFMnTqVUaNGdfr4W2+9lRtvvInhI0cx6KThjJ0wiaRYOzNPmc4b\nU6dwweypDBw4kFmzZh1+zC233MK8efPIzs7m448/Prx88uTJ3HjjjUyfPh2Am2++mZNPPlm6YIQQ\nRyVT/nYjrTV1rsAl/x4faXFRZCY42/1Vnt4U7s+rEKKFTPnbi/x+TWWDuWrU5fVhs1rIS40lIVp+\nakwIERoS7ieg+UckDlY34fH5iXFYGZgSQ2K0HYtcGSqECKE+F+5dmXe8L2iecbHe5SXabmVgchxx\nzj73dBKqbjchRGj1qTRyOp1UVFSQmtr+b2/2BY1uLyU1LmqaPNgsFnKSokmJ7ZtzuGitqaiowOl0\nhrooQohe1qfCPTc3l8LCQsrKykJdlHZVN3qobfJiURDvtBEbZaO0WlEa6oJ1wul0kpubG+piCCF6\nWZ8Kd7vdzpAhQ0JdjCNorXngzW08858Crj1lEHfNH0WCU06WCiH6rj4V7n2R1ppfvrOdZ/6zhxtP\nzeO+i8b0yS4YIYQI1qUrVJVS85RSO5RSu5VSd7dzf7JS6lWl1Eal1BdKqXHdX9Te5/drfvH2Np5c\nkc83ZgyWYBdChI2jhrtSygo8CswHxgBXK6XGtFntXmC91noCcD3wx+4uaG9r8vj4/vPreOrTPdww\nczA/WzBWgl0IETa60nKfDuzWWudrrd3A88DFbdYZA3wEoLXeDuQppTK6taS9qKrBzfV//YI3Nx7k\nnvmjuH/B2JBfZSqEEMeiK+GeAxQE3S4MLAu2AbgMQCk1HRgMHDFEQyl1i1JqjVJqTV8dEbPtYA2X\nPPof1hdU8eerT+bbZwyVFrsQIux016yQvwKSlFLrge8B6wBf25W01k9qradqraemp6d30667z6vr\nCrn0sf/Q4Pax+FuncNHE7FAXSfRlWoO74ejrhSOtYdsbUF8R6pKI49SV0TIHgIFBt3MDyw7TWtcA\nNwEo08zdA+R3Uxl7nMvr46G3tvHPlfuYPiSFR645mQHx3Xzhj98Huz+AoXPAKoOUwl7pNnjnLtjz\nCQycAeOvgDEXQ0waWLrQZtIaNi2B8h3gdYG3CepKoboAaopg8Cw4/2GISTn+MmoNngZwxB77Y3e8\nDS9cB8POgWtfgr767bV0O2x7HXa9D8mDYfQCGDYXHDGhLllrxZshbQTYHL22y6POCqmUsgE7gTmY\nUP8SuEZrvSVonSSgQWvtVkp9Czhda319Z9ttb1bIUCiobOC2xV+xsbCab50+hB/PG4W9sQK0D+Iz\nj21jnkZY9Rjs/ggufQKSgo6Jy38Ny38B17wII87r3krUlcHW12DTS6AssGhxx6FQtA62vApn3gv2\nDg5gWpufOMwYCxZr18uhNXz9IXzysCnHKbfAqIvMwUxrqDkA9pgTCywArxs2vQglW6Hya7Pdeb+G\nvFlHf6y7Hl7/HigrzP4hpI88tn1XFcDKR+GLJyEqHiYugvxPoGxbyzoWuwnU3Glw0pkw9CzzXAZb\n+w944/uAApvTfOhjB0Birnl+tr4OsWlwyePm8W35feag0MzqaN1oyF8OH/4cDnwFJ50BJ38DRl0A\n9uigbfhh7TOw8jGY/xsYPtcs9zTBo9PNwcbbCIueg1HnH9vzBOY17+yg4KqF8p2QM6XjdWqK4KWb\nzDrjr4Dsk6EyHza/3HJwBLP80D5orDTvseQh4EwAZyIMmglTbzJ/Hy+tob7cvOZtPzeuOnMw3Pgi\n7PvclHPOT83r11AJb/8INi+Bs/8HZv/o+MsQ0NVZIbs05a9S6nzgD4AVeEZr/ZBS6jsAWusnlFIz\ngX8AGtgCfFNrfaizbfaFcP9oewk/eH49GvjtlRM5b2wmHNoLT881LbDvruxai8XvM2+2D39uWl4W\nO2SMgZveNS2Igi/hmfPMAeO8X8LM75544evKYPubptWS/4nZdvpoE3YZY+H6pUe+mfd8Cs8tAned\nCfcz72p/26uegHfvgpm3w3kPda08Revhvf+BvZ9C0iAT7of2QuIgGDDKHFTqyyAqARb+ywTO8aj4\nGpb8Pzi4HmzRkDrUbNcaBd/93Hz4OtJYBYsXQuEX5rGeBhh3GUy6FpIGm2Bt74DXUAmf/xl2LoPS\nLYCCKTfC2f8Lsanmg1+yBb7+yGzT22Qes+9zqNhltjH92zDvl+ZgWbzJvMcGzYDrXmn/AFq0Hl75\nlgm/rEng95rtuhvAVWNew2AWu2kZZoyBuhLYswISckxLdsdbULUfohJh5DyzLHkwvP1j2P852GNN\nGW7+wBzsVjwMHz0I170My/4HPPVw2xetDwxg3vfb3jBlG36uCVO/zzQeVjxs9pk5AXImQ95pMPy8\nlgPQ/tWmflX7YOLV5uDiTDjyefj8z+Z9ZXWAz20OgPWl5jUYPAvGXgKjLzINMZ8X9v0Htr8F1YXm\neWqogNKtpu7TvgkzbzOhG6ymCHZ/CBW7zb+oBJh7P8QHxoS4G2Dpd029ABzxEJ0Mfo85wLpqzd+J\nAyF3qnlOHLEw9f/B+sWmDM5ESDnJPMcnqFvDvSeEOtwPVjdy1m+XMzQ9jsevncKg1BjzgfzrueYF\nRsM3P4CB09rfQG2xeSPtXGa+EjZWQtZEOPch8wFfvBDGXQ4X/QGeOM20khorTZCc/5uW7WgNn/8J\nRl4AacO6VviPHoRPfwfab1ooYy+B8VeaUN/xLrxwLWRPhm+80hJ2O5fBi9dDcp4Jsvzl5uCVOrT1\ntvf+B/5xkfmgNR4yH/BhczsvT305/Olk0wI948cw+QYTFjvegdVPmPuzT4bsSbDmb+b5veQxmHCV\nOQCs+ZsJ6+Q8SBlqWmnttcI3LYE3fmC2veDPMOpC0wWyfxU8M898mC78vVm3qQZW/MYcZAadCqnD\nYMmN5mv85U9D3umw8s+w+kkTXs3GXgZX/q31fl+9FTa+AINPNSE26oIjn7eOVBeagFr9hCnvhX8w\nB3p3PXznM4jr5NyTuwE++TWUbDbPrdVhGgtRieb1sUUBgcZH4yETYiVbTdDMugOmftMcrPx+c9Dd\n+IIJvqYq8xhnomlsDJkNT51tAmnhs+YzMGyu+XvPCvN+OONuOOuelrLlfwLv/aTlR+ytDjjpLPN6\nlu8wDY0hp8PBDXBwo/kGEJ9l3ht+L3z2exOGI+ebb0GJuXDZU+aAF+xv55vX8sY3TGju/tC8P8Zd\nDoltx3V0oGgdfPZ/5tuQMxHO+wVMusbct+5f8O494K41B8jkPNNAc8TCgkcgawI8d7Wp56nfM4+v\nLzMNBavdvC5RcebANfAU834s2wHv/Nh8xgaMhUsfh21vwqe/hR/nmwPDCZBwP4ofvrSB19cX8eGd\nZzAwJcYcgZ+9FAq/hIX/hpduMIG54E8tDypca0K1aB3UFpll0Skw/BzzwW0OG4AVv4WPHjChUpkP\nN75lvp4l5sI1L7Rs89Be+ONE09q9+aPOP+xgWie/GWJaQ+c+ZAK97beLrUvNV9nEHIjLMAeQg+sh\nY5xpKfo98Mg0s41vvNby+Joi+Mts8wa+6R34xwLT6rj1887L9dadJqBvWw1pwzsvf2OV6cvd+6n5\nMBR8YfafMc50rzRUAAru3NHScoJAgJ9n+rcvf7p1lxfAu/fCqkfh+tchIRuev8YcRJTV1BdMa33h\nv1q6H8CEYskWE8Lb3jDhF7xvnxceHgoj5sFlf+m8bp1Z9bgJEUesOfjf8GbXupG6m89jnvuDG02L\nubmeBV/A3y8AlHk9bvvCtO7BvJe2v2XCvWq/eb4KVptvZXPvMyG97XXzTdIRD7PvhNEXt3wWfF5z\nvunLp83/aNPImfcrc5BqbsXXFJn3UPOBs6HSPPen/xDO/smJ1710O7z5A9i/0hyIrA7YtQwGnwbz\nfw3po8w3i9Lt8MrNJtAd8eb5uPyvMOLcru9LayjeaLZpi2p5/175D9MYOwFdDXe01iH5N2XKFB0q\nWw5U67y739QPvrnFLKiv0HrxIq3vS9B640tm2Svf0fqhHK1ddea2u0Hr34/T+tcnab3km1p//qjW\n+7/Q2udtfyd+v9YvXG+2+cHPzbLnrtH6kVNar7frfbPOfQlaPzXH7KczBV+adTe93Pl6W9/Q+tnL\ntP7nJebfa7dp3VjVcv/qJ1vq63VrXbhG6yfP1vqhbK1Ltpl1ijdr/fN0s51tb2r91g/NOmv+3rKd\n0u1a35+s9Zt3dl6eYJ4mrV/5tta/HaX1hw9qXVXQct/XH5tybX+n9WM+edgsD65DMFe91n88Wevf\njjSv269P0jr/E7N8z6daf/ZHrQ981Xm5Sraafax+Mqg8y82yLUu7Xr+ObHlN6wezTFn6ovXPmbp+\n9IvWy6sPaP2LXHPfLwdp/eRZWn/6f1q7G499HxX55r3WVnWR1j9L0frde48sT3vrHy+fz7y+D2Vr\n/UCG1isfN8va8ri0fu+n5v1euv3E9+v1mOdw6e0nvClgje5CxvbLcL/u6VV6wv3LdFW9W+sd72r9\n8Ajzxlr1RMtKez4zb6x1i83t5b8xt/M/6fqOXPUmFLxuc/ude7R+MNMEf7NVf2kJlPsStH7xxvbf\nbM2aQ66urOvlaI/Pq/VfzjRh82Cm2eb9ySaAgjWX674Es+6fJrcOwH9fZd60J1qeZk21Wt+XqPXH\nv2y9/LlrzL47s2+l1vcnaf2XM1ofMI7Fn6dp/bcLWm6/9SOtHxjQcpA/Uc3vhb6qIr/1+7NZbalp\nBPWkF643B4/mBs4L3zCfzc4+D8ertkTrqsLu325nnrtG69+Pbf/5PQZdDff+MSavqQb2fgb5y6nb\nuZz7K2uIT0ohcfHD5uTagLFw7Yumz7zZ4FPNCZD1/zYn/z77vTlxM2R21/friIExC1puJw82X8nr\nyyBugFlWsdt89Zt2szlJ9sH9kDkeTv/v9re5Z4Xpwmh7UuhYWaxw8SNmOF/GWNPXOWjmkSOEpt1s\n+ggTsiEn8E3wxevh7R+ar5073zUnn060PM2i4syJwaJ1rZcXrYfBMzt/7KAZ8L2vTFltUce3/zEX\nm77RujJTp+1vwdCzj284YXusfXw20ZQOZmU9Wndhd5h2sxn1tfkVM+Jk90cw/vKuDS09Vs2fv940\nbI7puirfBekjenx3kR/uNQfhiVnQUIG2RbODUVTZ0zgrLcqE6ewfmyFxbcNAKdMv+NED8Oq3zSiA\ncx44sbIk55n/D+1rHe6pQ83+Zv3AhNjHvzD9+JnjWz/e02T6Oqd+88TK0SxjLNz4ZufrKGU+aMGu\n+qcJ+K/+afpbT7m1e8rTLHuSOWHXrK4UagrNqJGj6SicumrsJeZE7PY3zEngmkI4694T26bomrzT\nTB/1l0+sCLvxAAATsUlEQVSbRoa7FkbMD3Wpus/QOeb/rz/slXDvgUNiH/PB/Wao0rUv8+HFX3J5\n3Y+ovuBJLNe/Cje/b07UdNTKm3i1GW2xZ4UZQnWiwZEUOEF1aG/LsuZwBxOkF/zetJRfvdWM5w5W\n+IUZDncs3x56gs1hAn7WHXDpXzoeL3+8siZBXbEZkQTmgAcmbHvagDHmJPiW18wIB2UxIzpEz1PK\ntN6LvjKjhGzRxz9kti9KHmzeW7s/7JXdRXa4718NG5+HU7+HHjaHR1bsZ2BKNAu6Oq1AYo4Z+haf\n1XE3ybFIGmT+r9pr/ve6zLCr1KAhkLGpcNEfoWSTaUEG27PCjP4YfOqJl+VE2Rxwzs97ZsRHc4g3\nh/rB9YAyw9J6mlIw5pLA0MEXzVjqE73oSnTdhIVm3H3BatMd1nZsfbgberbpIg6+AK2HRG64+31m\nrGl8Npz236zMr2B9QRXfnj0Um/UYqn3ZU/DtFZ1fHNNVjhgzNPHQPnP70F4zVj21zfj2UeebLqFP\nfw+FQcNF8z8xwxfbu9gjkmSOB1Qg1DH972kjuuc16IoxF5vXpXq/Gd4qeo8zASYuNH+PnBfasvSE\noXPMmP/9K3t8V5Eb7uv+ZcLh3AcgKo7HPv6a9PgorphyjL8n6kzo3pMvSYNbumUqdpv/27sgZt4v\nA+O1rzVji121cGBt6LtkekPbk6pF60w/fG/JHG9OpsPxXXYvTsyp3zdX0Y6+KNQl6X55p5mLpXqh\nayYyw93TCB/+zIz+GHc5Gwqq+Gx3OTefNgSn/RjmSukJyYPNJdfQEu4p7YS7M9HMQ+NthH9dbvp/\nta9/hDuYMC9ab06I1x7snf72Zs0ntydf39KVJnpPyhBzdewJXsnZJ0XFwRV/NVdT97DIDPeKr82V\njtNuBqV4bPluEpw2rp0xONQlMy336kJzpWDFbjOHTXRS++tmjDGTNh3aB6/fbuZPGXhK75Y3VJpP\nqu5819zuzXAHmHKDmeJAiO425uITH5zRBZEZ7rUHzf+JuRRVNfLe1hJuODWPuKg+MPIzOS/Qn1to\nDkJt+9vbypsFlz9lziEMnB55J5g60twNs/bvZsRK22GhQohO9YG06wE1gXlfErJ5c2MRWnPsfe09\npXm+jqp9puU+7JyjP2bMxWY8etwxTkEczjIncPikavro7ruISIh+IrJb7nGZvLHhIBNzExmc2kfC\noXmse/FmMzVrV2cXzDut67NGRoLmk6rQ+10yQkSAyAz3miKITWdPlYdNB6r71s/lJeSAxWbm/oaj\nd8v0Z81dMxLuQhyzyA33+Cze2FCEUnDhhD4U7labmfZ333/MbQn3jjVPN9CbwyCFiBCR2edeexCd\nmMPrG4qYlpdCZmI3Xx5/ooLHuvfCWfOwNekaczI15+hTVwshWovYlnuVLZ3dpXV9q0umWfMEYokD\n+8/ol+MRnQQzvtMzswIKEeEi71PjaYLGSjbXxGK1KM4f1wdHmDSPmOnqyVQhhDhGkRfugZEyn5Xa\nmTUsjdS445zXuyc1j5iR/nYhRA+JvHAPjHHfWhfHvLF9sNUO5ketQcJdCNFjIi/cAy33Yp3C2Ow+\nOnti5ngzNUIkTowkhOgTIm+0TKDlXqxTGDYgLsSF6YDNARf8LtSlEEJEsIhsubuUk4TEFGL7wlwy\nQggRApEX7jVFlKlURmT20g87CCFEHxRx4a5rDlDgTWJEhoS7EKL/irhw91YVcVAn993+diGE6AWR\nFe5+P9b6Ekp0srTchRD9WmSFe0M5Fu3lYF8eKSOEEL0gssI9MAzSE5MpI2WEEP1aRIZ7dGof+dUl\nIYQIkYgKd38g3JMy+8APYQshRAhFVLjXlO7HpxXZ2YNCXRQhhAipiOqYri8voJFkhmUlh7ooQggR\nUhEV7r6qA5TpZIbLSBkhRD8XUd0y9oYSqm1pMlJGCNHvdSnclVLzlFI7lFK7lVJ3t3N/olLqDaXU\nBqXUFqXUTd1f1KNLcJfiie2jc7gLIUQvOmq4K6WswKPAfGAMcLVSakyb1W4DtmqtJwJnAr9TSjm6\nuayd8jXVEksD1sSc3tytEEL0SV1puU8Hdmut87XWbuB54OI262ggXimlgDigEvB2a0mP4mBBPgAx\naTLGXQghuhLuOUBB0O3CwLJgjwCjgSJgE3CH1trfLSXsooayvQA4U2WMuxBCdNcJ1fOA9UA2MAl4\nRCl1xG/cKaVuUUqtUUqtKSsr66ZdG/5D5vhjl3AXQoguhfsBYGDQ7dzAsmA3Aa9oYzewBxjVdkNa\n6ye11lO11lPT09OPt8ztUtUF+LTCmSLdMkII0ZVw/xIYrpQaEjhJugh4vc06+4E5AEqpDGAkkN+d\nBT0ae20BB0klPjamN3crhBB90lEHhGutvUqp24FlgBV4Rmu9RSn1ncD9TwAPAH9XSm0CFHCX1rq8\nB8t9BGdDEQU6jYlOGeMuhBBdSkKt9dvA222WPRH0dxFwbvcW7djENh6kmGGcYreGshhCCNEnRMYV\nqj4vce4yym0ZoS6JEEL0CZER7rVFWPFRZZerU4UQAiIl3Kv2A1DrzApxQYQQom+IkHA3Y9wbYmTq\nASGEgEgJ92oT7u7Y7BAXRAgh+obICPeq/ZSTRExsbKhLIoQQfUJkhHt1AQd0KvEyxl0IIYAICXd/\nVQEF/jQSnPZQF0UIIfqE8A93vx9VXUChTichWsJdCCEgEsK9vhTlc3NAp5Eg3TJCCAFEQrgHhkGa\ncJeWuxBCQCSEe7W5gOmATiMhWlruQggBkRDu0nIXQogjREC478dlT6COGDmhKoQQAeEf7tUF1EaZ\nOWWk5S6EEEb4h3tVAYfsGdgsCqc9/KsjhBDdIbzTUGuoLqDMOoCEaDtKqVCXSAgh+oTwDvfGQ+Cu\no0QNkDHuQggRJLzDvTpopIycTBVCiMPCO9zLdwGQ78+QScOEECJIeId78UawOtjmyZSRMkIIESTM\nw30TpI+i0iXDIIUQIlj4hrvWcHAjZE6gptErUw8IIUSQ8A33uhJoKMeXMY5Gj09a7kIIESR8w714\nEwD1yaMB5ISqEEIECeNw3wjAofiRADIUUgghgoRxuG+CpMFU62hATqgKIUSw8A73zPHUNHoBabkL\nIUSw8Ax3Vx1UfA2ZE6ht8gDS5y6EEMHCM9xLtwIasiZQEwh3abkLIUSL8Az3wMnUVt0y0nIXQojD\nwjTcN0F0MiTkUNPkwaIg1iHhLoQQzcI33DPHg1LUNHqId9qxWGQudyGEaBZ+4e7zQskWyJwAQG2T\nV06mCiFEG+EX7hW7wdtkWu5ATZNHxrgLIUQb4RfugWkHDoe7TBomhBBH6FK4K6XmKaV2KKV2K6Xu\nbuf+Hyml1gf+bVZK+ZRSKd1fXGDEuXD9UkgbAUjLXQgh2nPUcFdKWYFHgfnAGOBqpdSY4HW01g9r\nrSdprScB9wCfaK0re6LAOBPhpDPBagK9+YSqEEKIFl1puU8Hdmut87XWbuB54OJO1r8aeK47CtcV\ntU3SLSOEEG11JdxzgIKg24WBZUdQSsUA84CXT7xoR+fza2pdXumWEUKINrr7hOpFwH866pJRSt2i\nlFqjlFpTVlZ2wjura5JJw4QQoj1dCfcDwMCg27mBZe1ZRCddMlrrJ7XWU7XWU9PT07teyg4cnldG\nxrkLIUQrXQn3L4HhSqkhSikHJsBfb7uSUioROANY2r1F7Fh1Y/OMkNJyF0KIYEdt8mqtvUqp24Fl\ngBV4Rmu9RSn1ncD9TwRWvRR4T2td32OlbUNa7kII0b4upaLW+m3g7TbLnmhz++/A37urYF3R4PIB\nEBsl4S6EEMHC7wrVIA2e5nC3hrgkQgjRt4R1uDe6zWiZaJnuVwghWgnrcK8PdMvE2KXlLoQQwcI6\n3BsD3TIx0i0jhBCthHW4N7i9WC0KhzWsqyGEEN0urFOx3uUjxm5FKfkVJiGECBbW4d7o9kmXjBBC\ntCOsw73e7SVGRsoIIcQRwjrcG90+YhzSchdCiLbCOtwbJNyFEKJdYR7uXrmASQgh2hHm4e4jVlru\nQghxhLAP92gJdyGEOEKYh7tX+tyFEKIdYR7uPmKlz10IIY4QtuHu82tcXr90ywghRDvCNtwbAtP9\nSstdCCGOFLbh3ug2M0JKy10IIY4UtuHeEAh3OaEqhBBHCttwrw90y8jcMkIIcaSwDfdGabkLIUSH\nwjbc6yXchRCiQ2Eb7o3SLSOEEB0K23CXE6pCCNGxsA136ZYRQoiOhW24H+6WiZJuGSGEaCtsw725\nWybaLi13IYRoK6zD3Wm3YLWoUBdFCCH6nDAOd/lxbCGE6EgYh7tPumSEEKID4RvuLh+xURLuQgjR\nnvANd49PfhxbCCE6ELbh3uj2EiPdMkII0a6wDfd66ZYRQogOhW24N0q3jBBCdChsw73e5SVWph4Q\nQoh2dSnclVLzlFI7lFK7lVJ3d7DOmUqp9UqpLUqpT7q3mEdqdPvkJ/aEEKIDR+3XUEpZgUeBc4BC\n4Eul1Ota661B6yQBjwHztNb7lVIDeqrAAFprGjw+mTRMCCE60JWW+3Rgt9Y6X2vtBp4HLm6zzjXA\nK1rr/QBa69LuLWZrLq8fn1/LFapCCNGBroR7DlAQdLswsCzYCCBZKbVcKbVWKXV9dxWwPfITe0II\n0bnuavragCnAHCAaWKmUWqW13hm8klLqFuAWgEGDBh33zho8Eu5CCNGZrrTcDwADg27nBpYFKwSW\naa3rtdblwApgYtsNaa2f1FpP1VpPTU9PP94y0+CSn9gTQojOdCXcvwSGK6WGKKUcwCLg9TbrLAVO\nU0rZlFIxwCnAtu4tagv5iT0hhOjcUZu+WmuvUup2YBlgBZ7RWm9RSn0ncP8TWuttSql3gY2AH3ha\na725pwrdEu7SchdCiPZ0KR211m8Db7dZ9kSb2w8DD3df0TrW0PwTe9JyF0KIdoXlFarSLSOEEJ0L\ny3A/PBRSfhxbCCHaFZbhXt/cLSNT/gohRLvCMtybu2VkbhkhhGhfmIa7F6tFEWULy+ILIUSPC8t0\nbHD7iLFbUUqFuihCCNEnhWW4N7p9xMivMAkhRIfCMtzr3T65gEkIIToRluHe6PYSLSNlhBCiQ2EZ\n7g1u+XFsIYToTFiGe71bfhxbCCE6E5bh3uj2ygVMQgjRibAM9wYZLSOEEJ0K33CXq1OFEKJDYRru\nXmKlz10IIToUduHu82uaPH6ZV0YIIToRduHeKD+OLYQQRxV24d7yK0zSLSOEEB0Jv3B3SctdCCGO\nJvzCXX5iTwghjirswr3RI90yQghxNGEX7vXSLSOEEEcVduHe0i0jLXchhOhI2IV7eryD+eMySYl1\nhLooQgjRZ4Vd83fK4BSmDE4JdTGEEKJPC7uWuxBCiKOTcBdCiAgk4S6EEBFIwl0IISKQhLsQQkQg\nCXchhIhAEu5CCBGBJNyFECICKa11aHasVBmw7zgfngaUd2NxwkV/rHd/rDP0z3r3xzrDsdd7sNY6\n/WgrhSzcT4RSao3Wemqoy9Hb+mO9+2OdoX/Wuz/WGXqu3tItI4QQEUjCXQghIlC4hvuToS5AiPTH\nevfHOkP/rHd/rDP0UL3Dss9dCCFE58K15S6EEKITYRfuSql5SqkdSqndSqm7Q12enqCUGqiU+lgp\ntVUptUUpdUdgeYpS6n2l1K7A/8mhLmt3U0pZlVLrlFJvBm73hzonKaWWKKW2K6W2KaVm9pN6/1fg\n/b1ZKfWcUsoZafVWSj2jlCpVSm0OWtZhHZVS9wSybYdS6rwT2XdYhbtSygo8CswHxgBXK6XGhLZU\nPcIL3Km1HgPMAG4L1PNu4EOt9XDgw8DtSHMHsC3odn+o8x+Bd7XWo4CJmPpHdL2VUjnA94GpWutx\ngBVYROTV++/AvDbL2q1j4DO+CBgbeMxjgcw7LmEV7sB0YLfWOl9r7QaeBy4OcZm6ndb6oNb6q8Df\ntZgPew6mrv8IrPYP4JLQlLBnKKVygQuAp4MWR3qdE4HZwF8BtNZurXUVEV7vABsQrZSyATFAERFW\nb631CqCyzeKO6ngx8LzW2qW13gPsxmTecQm3cM8BCoJuFwaWRSylVB5wMrAayNBaHwzcVQxkhKhY\nPeUPwI8Bf9CySK/zEKAM+FugO+pppVQsEV5vrfUB4LfAfuAgUK21fo8Ir3dAR3Xs1nwLt3DvV5RS\nccDLwA+01jXB92kzzClihjoppS4ESrXWaztaJ9LqHGADJgOPa61PBupp0xURifUO9DNfjDm4ZQOx\nSqnrgteJxHq31ZN1DLdwPwAMDLqdG1gWcZRSdkyw/1tr/UpgcYlSKitwfxZQGqry9YBZwAKl1F5M\nd9vZSql/Edl1BtM6K9Rarw7cXoIJ+0iv91xgj9a6TGvtAV4BTiXy6w0d17Fb8y3cwv1LYLhSaohS\nyoE5+fB6iMvU7ZRSCtMHu01r/fugu14Hbgj8fQOwtLfL1lO01vdorXO11nmY1/UjrfV1RHCdAbTW\nxUCBUmpkYNEcYCsRXm9Md8wMpVRM4P0+B3NuKdLrDR3X8XVgkVIqSik1BBgOfHHce9Fah9U/4Hxg\nJ/A18JNQl6eH6nga5qvaRmB94N/5QCrm7Pou4AMgJdRl7aH6nwm8Gfg74usMTALWBF7v14DkflLv\nnwHbgc3As0BUpNUbeA5zTsGD+Zb2zc7qCPwkkG07gPknsm+5QlUIISJQuHXLCCGE6AIJdyGEiEAS\n7kIIEYEk3IUQIgJJuAshRASScBdCiAgk4S6EEBFIwl0IISLQ/wfyL4AstkTMIAAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c714a4fda0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.legend(['training', 'validation'], loc = 'upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 3s     \n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.86876\n"
     ]
    }
   ],
   "source": [
    "print('Test accuracy: ', results[1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
